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Abstract 


We consider a problem of atomic broadcast in a dynamic setting where processes may join, 
leave voluntarily, or fail (by stopping) during the course of computation. We provide a formal 
definition of the Dynamic Atomic Broadcast problem and present and analyze a new algorithm 
for its solution in a synchronous system, where processes have approximately synchronized 
clocks. 

Our algorithm exhibits constant message delivery latency in the absence of failures, even 
during periods when participants join or leave. To the best of our knowledge, this is the first 
algorithm for totally ordered multicast in a dynamic setting to achieve constant latency bounds 
in the presence of joins and leaves. When failures occur, the latency bound is linear in the 
number of actual failures. 

Our algorithm uses a solution to a variation on the standard distributed consensus problem, 
in which participants do not know a priori who the other participants are. We define the 
new problem, which we call Consensus with Unknown Participants, and give an early-deciding 
algorithm to solve it. 


1 Introduction 


We consider a problem of atomic broadcast in a dynamic setting where an unbounded number of 
participants may join, leave voluntarily, or fail (by stopping) during the course of computation. 
We formally define the Dynamic Atomic Broadcast (DAB) problem, which is an extension of the 
Atomic Broadcast problem [17] to a setting with infinitely many processes, any finite subset of 
which can participate at a given time. Just as Atomic Broadcast is a basic building block for state 
machine replication in a static setting [20, 27], DAB can serve as a building block for state machine 
replication among a dynamic set of processes. 

We present and analyze a new algorithm, which we call Atom, for solving the DAB problem in 
a synchronous crash failure model. Specifically, we assume that the processes solving DAB have 
access to approximately-synchronized local clocks and to a lower-level dynamic network that guar- 
antees timely message delivery between currently active processes. The challenge is to guarantee 
consistency among the sequences of messages delivered to different participants, while still achieving 
timely delivery, even in the presence of joins and leaves. 

Atom exhibits constant message delivery latency in the absence of failures, even during periods 
when participants join or leave; this is in contrast to previous algorithms solving similar problems 
in the context of view-oriented group communication, e.g., [1, 9]. When failures occur, Atom’s 
latency bound is linear in the number of failures that actually occur; it does not depend on the 
number of potential failures, nor on the number of joins and leaves that occur. 

A key difficulty for an algorithm solving DAB is that when a process fails, the network does 
not guarantee that the surviving processes all receive the same messages from the failed process. 
But the strong consistency requirements of DAB dictate that processes agree on which messages 
they deliver to their clients. The processes carry out a protocol to coordinate message delivery, 
which works roughly as follows: Each Atom process divides time into slots, using its local clock, 
and assigns each message sent by its client to a slot. Each process delivers messages to its client 
in order of slots, and within each slot, in order of sender identifiers. Each process determines the 
membership of each slot, and delivers messages only from senders that it considers to be members 
of the slot. To ensure consistency, the processes must agree on the membership of each slot. 

Processes joining (or voluntarily leaving) the service coordinate their own join (or leave) by 
selecting a join-slot (or leave-slot) and informing the other processes of this choice, without delaying 
the normal delivery of messages. When a process fails, Atom uses a novel distributed consensus 
service to agree upon the slot in which it fails. The consensus service required by Atom differs from 
the standard stopping-failure consensus services studied in the distributed algorithms literature 
(see, e.g., [21]) in that the processes implementing the consensus service do not know a priori who 
the other participants are. Atom tracks process joins and leaves, and uses this information to 
approximate the active set of processes that should participate in consensus. However, different 
processes running Atom may have somewhat different perceptions of the active set, e.g., when a 
participant joins or leaves Atom at roughly the time consensus is initiated. 

In order to address such uncertainties, we define a new consensus service, consensus with un- 
known participants (CUP). When a process i initiates CUP, it submits to CUP a finite set W; 
estimating the current world, in addition to 7’s proposed initial consensus value v;. The worlds 
suggested by different participants do not have to be identical, but some restrictions are imposed 
on their consistency. Consider, e.g., the case that process k joins Atom at roughly the time CUP 
is initiated. One initiator, 7, may think that k has joined in time to participate and include & in 
W;, while another, 7, may exclude k from W;. Process k cannot participate in the CUP algorithm 
in the usual way, because 7 would not take its value into account. On the other hand, if k does not 


participate at all, 2 could block, waiting forever for a message from k. We address such situations 
by allowing k to explicitly abstain from an instance of CUP, ie., to participate without providing 
an input. A service that uses CUP must ensure that for every 7, (1) W; includes all the processes 
that ever initiate this instance of CUP (unless they fail or leave prior to 7’s initiation); and (2) if 
j € Wij, (and neither i nor j fail or leave), then j participates in CUP either by initiating or by 
abstaining. Thus, W; sets can differ only in the inclusion of processes that abstain, leave, or fail. 

Note that once an instance of CUP has been started, no new processes (that are not included 
in W;) can join the running instance. Nevertheless, CUP provides a good abstraction for solving 
DAB, because Atom can invoke multiple instances of CUP with different sets of participants. 

We give an early-deciding algorithm to solve CUP in a fail-stop model [26], that is, in a time- 
free crash failure model where processes are equipped with perfect failure detectors [5]. The failure 
detector is external to CUP; it is implemented by Atom. CUP uses a strategy similar to previous 
early-deciding algorithms for consensus with a predetermined set of participants [13], but it also 
tolerates uncertainty about the set of participants, and moreover, it allows processes to leave 
voluntarily without incurring additional delays. The time required to reach consensus is linear in 
the number of failures that actually occur during an execution, and does not depend on an upper 
bound on the number of potential failures, nor on the number of processes that leave. 

We also analyze the message-delivery latency of Atom under different failure assumptions. We 
show a constant latency bound for periods when no failures occur, even if joins and leaves occur. 
When failures occur, the latency is proportional to the number of actual failures. This is inevitable: 
atomic broadcast requires a number or rounds that is linear in the number of failures (see [2]). 

We envision a service using Atom, or a variation of it, deployed in a large LAN, where latency 
is predictable and message loss is bounded. In such settings, a network with the properties we 
assume can be implemented using forward error correction (see [3]), or retransmissions (see [28]). 
The algorithm can be extended for use in environments with looser time guarantees, e.g., networks 
with differentiated services; we outline ideas for such an extension in Section 7.7. 

In summary, this paper makes the following main contributions: (1) the definitions of two new 
problems for dynamic networks, expressed by the DAB and CUP services; (2) an early-delivery DAB 
algorithm, Atom, which exhibits constant latency in the absence failures; (3) a new early-deciding 
algorithm for solving CUP in a fail-stop model; and (4) the analysis of Atom’s message-delivery 
latency under various failure assumptions. 

The rest of this paper is organized as follows: Section 2 discusses related work. In Section 3, 
we specify the DAB service. In Section 4 we specify CUP and in Section 5, we present the CUP 
algorithm and its analysis. We then turn to the presentation of Atom: Section 6 specifies the 
environment and model assumptions for Atom, and Section 7 contains a detailed presentation of 
the Atom algorithm and its analysis. Section 8 concludes the paper. The Appendix contains 
rigorous correctness proofs for both CUP and Atom. 


2 Related Work 


A dynamic universe, where processes join and leave, was first considered in the context of view- 
oriented group communication work [7], pioneered by the Isis [4] system. The first analysis of 
time bounds of message delivery in synchronous group communication systems was performed by 
Cristian [9]. Our service resembles the services provided by group communication systems; although 
we do not export membership to the application, it is computed, and would be easy to export. 
View-oriented group communication systems, including systems designed for synchronous sys- 
tems and real-time applications (e.g., Cristian’s [9], xAMp [25], and RTCAST [1]), generally run 


a group membership protocol every time a process joins or leaves, and therefore delay message 
delivery to all processes when joins or leaves occur. Cristian’s system uses an atomic broadcast 
primitive to agree upon group membership. Since, unlike CUP, the atomic broadcast service works 
with a static universe, a process join has to be agreed upon before any new membership change 
is handled (voluntary leaves are not considered). Therefore, Cristian’s service exhibits constant 
latency only in periods in which no joins or failures occur. Latency during periods with multiple 
joins is not analyzed. xAMp is a group communication system supporting a variety of communica- 
tion primitives for real-time applications. The presentation of xAMp in [25] focuses on the various 
communication primitives and assumes that a membership service is given. The delays due to fail- 
ures and joins are incurred in the membership part, which is not described or analyzed. RTCAST 
is a real-time group communication system, for which a detailed analysis of membership latency 
was conducted [1]. The latency bound achieved by RTCAST is linear in the number of processes, 
even when no process fails, due to the use of a logical ring. Moreover, RTCAST makes stronger 
assumptions about its underlying network than we do — it uses an underlying reliable broadcast 
service that guarantees that correct processes deliver the same messages from faulty ones; the cost 
of this primitive is not considered in the analysis. 


Some group membership services avoid running the full-scale membership for join and leaves 
by using light-weight group membership [15] services; they use an atomic broadcast service to 
disseminate join and leave messages in a consistent manner, without running the full-scale group 
membership algorithm. However, unlike our CUP service, the atomic broadcast service such systems 
use do not tolerate uncertainty about the set of participants. Therefore, a race condition between 
a join and a concurrent failure can cause such light-weight group services (e.g., [23, 12, 15]) to 
violate the semantics of the underlying heavy-weight membership services. Those light-weight 
group services that do preserve the underlying heavy-weight membership semantics (e.g. [24]), do 
incur extra delivery latencies whenever joins and leaves occur. 


Other work on group membership in synchronous and real-time systems, e.g., [19, 18] has focused 
on membership maintenance in a static, fairly small, group of processes, where processes are subject 
to failures but no new processes can join the system. Likewise, work analyzing time bounds of 
synchronous atomic broadcast, e.g. [16, 10, 8], considered a static universe, where processes could 
fail but not join. Thus, this work did not consider the DAB problem. 


In a previous paper [3], we considered a simpler problem of dynamic totally ordered broadcast 
without all or nothing semantics. For this problem, the linear lower bound does not apply, and we 
exhibited an algorithm that solves the problem in constant time even in the presence of failures. 


Recent work [22, 6] considers different services, including (one shot) consensus, for infinitely 
many processes in asynchronous shared memory models. Chockler and Malkhi [6] present a con- 
sensus algorithm for infinitely many processes using a static set of active disks, a minority of which 
can fail. This differs from the model considered here, as in our model all system components may 
be ephemeral. Merritt and Taubenfeld [22] study consensus under different concurrency models; in 
their terminology, our model assumes unbounded congruency and [1, oo|-participation, which means 
that at least one process must participate and there is no bound on the number of participants. 
They show that with these assumptions, in an asynchronous shared memory model, infinitely many 
bits are required in order to solve consensus. The algorithms they give are not fault tolerant (they 
tolerate only initial failures). To the best of our knowledge, atomic broadcast has not been con- 
sidered in a similar context. Moreover, these problems were not considered in message-passing 
models, and it is not clear that a canonical transformation from the shared memory model the 
message-passing model applies to a setting with infinitely many processes. 


3 Dynamic Atomic Broadcast Service Specification 


We now present the DAB service specification. Our universe consists of an infinite ordered set of 
endpoints, J. The specification of DAB is parameterized by a message alphabet, M. The signature 
of the DAB(M) service is presented in Figure 1. 


Input: 
join,;, leave;, fail,, iel 
mcast;(m), meM, icel 


Output: 


join_OK;, leave_OK;, i¢I 


rev; (m), meM, ic€l 


Figure 1: The signature of the DAB(M) service. 


We do not consider recoveries from failure or rejoining after leaving. In other words, there 
cannot be multiple “incarnations” at a single endpoint. Instead of new incarnations, consider the 
same client joining at new endpoints. 


Assumptions about the application: DAB(M) assumes that its application satisfies the fol- 
lowing safety conditions: 


e For each i € I: 


— At most one join; and at most one leave; occur. 
— If leave, occurs, then it is preceded by join_0K;. 


— Any mcast;(m) has a preceding join_0K; but no preceding leave; or fail;. 


e At most one mcast(m) occurs for each particular m. 


DAB guarantees: Given an application that satisfies the above constraints, DAB(M) satisfies 
the properties we now specify. 

We first specify some basic integrity properties, both safety and liveness. We later specify the 
properties related to the ordering and reliability of messages. 

Basic safety properties: 


e Join/leave integrity: For each i: 


— At most one join_OK; and at most one leave_OK; occur. 
— If join_OK; occurs then it is preceded by joinj. 
— If leave_OK; occurs then it is preceded by leave,. 

e Message integrity: 


— No two rev;(m) actions occur for the same m and j. 


— If rcv; (m) occurs for some j then it is preceded by mcast;(m) for some 7. 


Basic liveness properties: 


e Eventual join: If join; occurs then either fail; or join_OK; occurs. 


e Eventual leave: If leave; occurs then either fail; or leave_OK; occurs. 


To specify the ordering and reliability guarantees of DAB, we require that there be a total 
ordering S on all the messages received by any of the endpoints, such that for all 7 € J, the 
following properties are satisfied. 

Safety properties: 


e Multicast order: If mcast;(m) occurs before mcast;(m’), then m precedes m’ in S. 
e Receive order: If rcv;(m) occurs before rcv;(m’) then m precedes m’ in S. 


e Multicast gap-freedom: If mcast;(m), mcast;(m’), and mcast,;(m’’) occur, in that order, and 
S contains m and m”, then S also contains m’. 


e Receive gap-freedom: If § contains m, m’, and m”, in that order, and rcv;(m) and rcv; (m’’) 
occur, then rcv;(m’) also occurs. 


Liveness property: 


e Multicast liveness: If mcast;(m) occurs and no fail; occurs, then S contains m. 


e Receive liveness: If S contains m, m is sent by 7 and 7 does not leave or fail, then rcv; (m) 
occurs, and for every m’ that follows m in S, rcv;(m’) also occurs. 


4 Consensus with Unknown Participants — Specification 


In this section we define the problem of Consensus with Unknown Participants (CUP). CUP is 
an adaptation of the problem of fail-stop uniform consensus to a dynamic setting in which the 
set of participants is not known ahead of time, and in which participants can leave the algorithm 
voluntarily after initiating it. Moreover, participants are not assumed to initiate at the same time. 
CUP uses an underlying reliable network, and a perfect failure detector. 

We begin with a description of CUP’s external signature (interface). We then specify the 
assumptions that CUP makes about its environment, including the application, the underlying 
network, and the external failure detector. We separate these into safety and liveness assumptions. 
Finally, we specify CUP’s safety and liveness guarantees. CUP’s safety guarantees depend on only 
the safety assumptions, that is, they are not allowed to be violated even if the liveness assumptions 
do not hold. On the other hand, CUP’s liveness guarantees depend on both the safety and liveness 
assumptions. 


4.1 External Signature 
The CUP specification uses the following data types: 


e J, an infinite ordered set of endpoints. Each endpoint in J corresponds to a potential partic- 
ipant in CUP. 


e V, a totally ordered set of values. Initial values and decision values are elements of V. 


Input: 
init,;(@v,W), ve V, WCI, W finite, i¢ 1  //i initiates with value v,world W 


abstain,;, ie I // i abstains 
net_rcv;(m), m€ Mcoyp, iE I // i receives message m 
leave;, ie! // i leaves 
leave_detect,(j), j, iE I // i detects that j has left 
fail,, iel // i fails 
fail_detect,(j), j, iE I // i detects that j has failed 
Output: 

decide; (v), vEV, ie! // i decides on value v 
net_mcast;(m), m€Mcyp, i€ I // i multicasts m 


Figure 2: The signature of CUP. 
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Figure 3: Interface diagram for CUP. 


e Mcup, a message alphabet. 


The external signature of CUP is presented in Figure 2, and depicted in Figure 3. 

The interface describes four kinds of interaction: “normal” interaction with clients of the CUP 
service, interaction with a multicast network, communication involving leaves and leave detection, 
and communication involving failures and failure detection. 


Normal interaction with clients: A process may participate in the CUP service in two ways: 
it may provide an initial value, in which case we say that the process initiates CUP, or it may 
decline to provide an initial value, in which case we say that it abstains. Participant 7 € I initiates 
CUP using the init;(v,W) action. Here, v is 2’s initial value, and W is its initial world, that is, 
the set of processes that 7 expects to participate in CUP. Participant 7 abstains using the abstain; 
action. Informally speaking, a participant abstains when it does not need to participate in CUP, but 
because of uncertainty about CUP participants, some other participant may expect it to participate. 


An environment assumption ensures that, if any process expects i to participate in CUP, 7 will in 
fact participate, unless it leaves or fails. CUP reports the consensus decision value to process i 
using the decide;(v) action. 


Multicast network: The network interface consists of the net_mcast and net_rcv actions. 


Leaves: A participant can leave the CUP service voluntarily using the leave; action. We assume 
that the environment provides a leave detector: the leave_detect;(j) action is used to notify i 
that j has left the algorithm voluntarily. 


Failures: The fail; action represents the failure of endpoint 7. We assume that the environment 
provides a failure detector, which uses the fail_detect;(j) action to notify ¢ that 7 has failed. 


4.2 Environment Assumptions 


Here we list and explain the assumptions that CUP makes about its environment. We classify these 
as safety and liveness assumptions. Formally, each of the properties given here is a trace property 


(21, Ch. 8]). 


4.2.1 Safety assumptions 


The first assumption expresses simple well-formedness conditions saying that each participant be- 
gins participating (by initiating or abstaining) at most once, leaves at most once, and fails at most 
once. 


e Well-formedness: For any i € I, 


1. At most one init; or abstain; event! occurs. 

2. At most one leave; event occurs. 

3. At most one fail; event occurs. 

4. No leave; or fail; precedes an init;. 
The next assumption says that, while the worlds W suggested by different participants in their init 
events do not have to be identical, CUP’s environment must guarantee that they have a certain 
kind of consistency. Namely, each W set submitted by an initiating participant z must include all 


participants that ever initiate CUP and that do not leave or fail prior to the init; event. This 
implies that every participant must be included in its own estimated world. 


e World consistency: If init;(*, W) and init;(*,*) events occur, then either 7 ¢ W, or a 
leave; or fail; event occurs before the init;(*, W) event. 


The next property describes the correctness of the message deliveries: every message that is received 
was previously sent, and no message is received at the same location more than once. Moreover, 
the order of message receipt between particular senders and receivers is FIFO. 


1An “event” is an occurrence of an action in a sequence. 


e Message integrity: There is a mapping from net_rcv events to preceding net_mcast events, 
such that the same message in Mcy p appears in both events, and such that no two net_rcv; 
events for the same 7 map to the same net_mcast event. Moreover, two net_rcv; events that 
map to net_mcast events of the same sender occur in the same order as the net_mcast events. 


The next two properties describe assumptions about leaves and leave detection. The first says that 
leave detection is “accurate”, in the sense that the occurrence of a leave_detect;(j) implies that 
j has really left; it also includes a simple well-formedness condition. The second property says that 
leaves are handled gracefully, in the sense that the occurrence of a leave_detect,;(j) implies that 
z has already received any network messages sent by j prior to leaving. Thus, a leave_detect;(j) 
is an indication that 7 has not lost any messages from 7. 


e Accurate leave detector: For any 7,7 € J, at most one leave_detect;(j) event occurs, and if 
leave_detect;(j) occurs, then it is preceded by a leave;. 


e Lossless leave: Assume net_mcast;(m) occurs and is followed by a leave;. Then if a 
leave_detect;(j) occurs, it is preceded by net_rcv;(m). 


The final safety assumption says that failure detection is accurate. 


e Accurate failure detector: For any 1,7 € J, at most one fail_detect,;(j) event occurs, and if 
fail_detect;(j) occurs, then it is preceded by a fail,. 


Note that we do not have a failure assumption analogous to the lossless leave property; thus, failures 
are different from leaves in that we allow the possibility that some messages from failed processes 
may be lost. 


4.2.2 Liveness assumptions 


The first liveness assumption says that, if any process 7 expects another process j to participate, 
then j will actually do so, unless either 7 or 7 leaves or fails. 


e Init occurrence: If an init;(*,W) event occurs and j € W, then an init;, abstain;, leave;, 
fail;, leave,;, or fail; occurs. 


The next assumption describes reliability of message delivery. It says that any message that 
is multicast by a non-failing participant that belongs to any of the W sets submitted to CUP, is 
received by all the non-leaving, non-failing members of all those W sets. 


e Reliable delivery: DefineU = Uger{ W | init,(*, W) occurs}. Ifi,7 € Vand net_mcast;(m) 
occurs after an init; or abstain; event, then a net_rcv;(m), leave;, fail;, or fail; occurs. 


The final liveness assumption says that the leaving or failure of any process that belongs to an 
initiator’s W set is detected by that initiator, unless it finishes by deciding, leaving, or failing. 


e Complete leave and failure detector: If init;(*,W) occurs, 7 € W, and leave; or fail; 
occurs, then fail_detect;(j), leave_detect;(j), decide;, leave;, or fail; occurs. 


4.3. CUP Service Guarantees 


Now we list CUP’s service guarantees. Again, we classify these as safety and liveness properties. As 
we noted earlier, CUP’s safety guarantees depend only on its safety assumptions, whereas CUP’s 
liveness guarantees depend on both its safety and liveness assumptions. 

Formally, each individual property is a trace property. The complete specification consists of 
two general trace properties whose respective sets of traces are defined by the following predicates: 


1. The conjunction of all the CUP safety assumptions implies all the CUP safety guarantees. 


2. The conjunction of all the CUP safety and liveness assumptions implies all the CUP liveness 
guarantees. 


4.3.1 Safety guarantees 


The first guarantee expresses well-formedness conditions saying that only participants that have 
initiated can decide, and each participant decides at most once. 


e Well-formedness: For any i € I, 


1. If decide; occurs then it is preceded by an init,. 


2. At most one decide; occurs. 


The next two guarantees are the main agreement and validity guarantees for consensus. The 
uniform agreement property says that everyone who decides agrees. The validity property has two 
parts: it says that any decision value is some participant’s initial value, and moreover, that any 
participant’s decision is no greater than its initial value. The latter is not a “standard” property 
for consensus but is needed for our use in Atom. 


e Uniform Agreement: For any i,j € I, if decide;(v) and decide,;(v’) both occur then v = v’. 
e Validity: For any i € I, if decide;(v) occurs then 
1. For some j, init;(v, *) occurs. 


2. If init;(v’,*) occurs then v < v’. 


4.3.2 Liveness guarantees 


CUP provides one liveness guarantee, which says that any participant that initiates and neither 
leaves nor fails must eventually decide. We do not make such a guarantee for a participant that 
abstains, that is, participants that abstain need not be informed of the decision value. 


e Termination: If an init; event occurs then a decide;, leave,;, or fail; occurs. 


5 The CUP Algorithm 


In this section, we present our implementation of CUP. 


5.1 Modeling Assumptions and Conventions 


We use the I/O automaton model of Lynch and Tuttle (see, e.g., [21, Ch. 8]), using standard 
precondition/effect (guarded command) pseudo-code, augmented with one new construct: effects 
may include statements of the form trigger (a), where a is an output action. Formally, we assume 
the automaton’s state contains a special FIFO buffer trigger-buffer. The trigger (a) statement 
adds a to the end of trigger-buffer. The action at the head of trigger-buffer is always 
enabled, and gets removed from trigger-buffer when it is performed. No other state changes are 
associated with action a. 

The fail; action described in the CUP interface represents the failure of endpoint 7. In terms 
of the algorithm, we interpret this to mean that once fail; occurs, i performs no more locally 
controlled actions, and input actions have no effect on the state. We treat this as a general 
convention, and do not include event handlers for fail; actions in our pseudo-code. 


5.2. The Algorithm 


Figures 4 and 5 contain the CUP implementation for a particular endpoint 7 € J. The algorithm 
includes no internal actions. Therefore, the signature consists of the actions indexed by this partic- 
ular 7 in the external signature of CUP (see Section 4). The message alphabet Mcyp is specialized 
to the set of messages of the following forms: 


e (i,r,v,W), where? € I,r EN, v € V, and Wis a finite subset of I. 
e (i1,0UT,r), wherei€ ITandr EN, 


The algorithm proceeds in asynchronous rounds numbered 1,2,.... In each round, a process 
sends its current estimates of the value and the world (the set of active processes) to the other 
processes. Each process maintains two-dimensional arrays, value and world, in which it collects 
the value and world information it receives from all processes in all rounds. It records, in a 
variable out[r], the other processes that it knows will not participate in round r because they 
have previously left, abstained, or decided. It also records, in a variable failed, the processes that 
it knows have failed. 


mode € { Ll, running, done}, initially 1 
round € N, initially 0 
for each r € N™, j € TI: 
value[r,j] € VU{ Ll}, initially 1 
world[r,j], a finite subset of I or 1, initially 1 
for each r € NT 
out[r], a finite subset of I, initially {} 
failed[r], a finite subset of I, initially {} 


Derived variables: 


for each r € Nt 
out-by[r], a finite subset of I, defined as Unt oy out [r’] 


failed-by[r], a finite subset of I, defined as Uns failed[r'] 


<r 


Figure 4: CUP; state. 
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init, (v,W) 
Eff: if mode = 1 then 
mode ¢ running 
round ¢ 1 
trigger (net_mcast, (i,1,v,W)) 


net_mcast, (i,r,v,W) where r > 2 
Pre: mode = running 
r = round + 1 
W = world[round,i] \ out[round] \ failed[round] 
// All messages for the previous round have been received. 
Vj € W: value[round,j] #1 
WA{}A v =mint{value[round,j] | j € Wh 
// No decision can be made. 
4 V j € world[round,i] \ out [round]: 
value[round,j] = value[round,i] A world[round,j] C world[round, i] 
Eff: round ¢ r 


net_rcv, (j,r,v,W) 

Eff: if mode # done A j ¢ failed-by[r] then 
value[r,j] ¢«v 
world[r,j] «+ W 


abstain, 
Eff: if mode = 1 then 
mode ¢ done 


trigger (net_mcast, (i,0UT)) 


decide, (v) 
Pre: mode = running 
value[round,i] # L 
VY j € world[round,i] \ out[round]: 
value[round,j] =v A world[round,j] C world[round,i] 
Eff: mode < done 
trigger (net_mcast, (i, 0UT)) 


net_rcv, (j,0UT) 

Eff: if mode # done then 
let r=min {r’ € Nt | value[r’,j] = 1} 
out[r] ¢ out[r] U {j} 


leave; 


Eff: mode ¢ done 

leave_detect, (j) 

Eff: if mode # done then 
let r =min {r’ € Nt | value[r’',j] = 1} 
out[r] ¢ out[r] U {j} 


fail_detect, (j) 

Eff: if mode # done then 
let r=min {r’ € Nt | value[r’,j] = 1} 
failed[r] ¢ failed[r] U {j} 


Figure 5: CUP; transitions. 
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The code works as follows. When an init; (v,W) input occurs, process 7 triggers a net_mcast (i,1,v,W) 
to send its initial value v and estimated world W to all processes, including itself. 

For each round r > 2, process 2 performs an explicit net_mcast;(i,r,v,W) to multicast its 
round r value v and world W. The world W is determined to be the set of processes that 7 thinks are 
still active, that is, the processes in 2’s previous world that 7 does not know to be out or to have 
failed in round r. Process i may perform this multicast only if its round is r-1, it has received 
round r-1 messages from all the processes in W, and it is not currently able to decide. The value v 
that is sent is the minimum value that 7 has recorded for round r-1 from a process in W. 

When a net_rcv;(j,r,v,W) occurs, process 2 puts v and W into the appropriate places in the 
value and world arrays. 

When an abstain; input occurs, process 7 sends an OUT message, so that other processes will 
know not to wait for further messages from it, and stops participating in the algorithm. 

Process 7 can decide at a round r when it has received messages from all processes in its 
world[r,i] except those that are out at round r, such that all of these messages contain the 
same value and contain worlds that are subsets of world[r,i]. The subset requirement ensures 
that processes in world[r,i] will not consider values from processes outside of world[r,i] in 
determining their values for future rounds. When process 7 decides, it multicasts an OUT message 
and stops participating in the algorithm. 

When a net_rcv;(j,OUT) occurs, process ¢ records that j is out of the algorithm starting from 
the first round for which 7 has not yet received a regular message from 7. 

When leave; occurs, process 7 just stops participating in the algorithm. When leave_detect;(j) 
occurs, process 7 records that j is out; when this occurs, the lossless leave assumption ensures that 
z has already received all the messages 7 sent. The round that is recorded for the leave is the first 
round after the round of the last message received from j. 

Process 7 knows that another process has failed if it learns about the failure via a fail_detect 
event. 

In the next section, we prove the algorithm’s correctness. In Section 5.3, we show that the 
algorithm is early-deciding in the sense that the number of rounds it executes is proportional to 
the number of actual failures that occur, and does not depend on the number of participants or on 
the number of processes that leave. 


5.3 The Early-Deciding Property 


We now show that the algorithm is early-deciding in the sense that the number of rounds it executes 
is proportional to the number of actual failures that occur, and does not depend on the number of 
participants or on the number of processes that leave. 

We start with some more lemmas. 


Lemma 5.1 Jf init;(*,W) occurs prior to init,;, then j € W. 

Proof: The environment well-formedness assumption implies that 7 does not leave or fail before 
it initiates, and hence does not leave or fail before 7 initiates. Therefore, by world consistency, 7 € 
W. | 


Invariant 5.1 Jf (i,1,*,W) and (j,2,*,*) are in the Net then] € W. 


Proof: By strong induction. For the inductive step, assume that, in the final state of the exe- 
cution, (i,1,*,W) and (j,2,*,*) are in the Net. Then both init; and init; events appear in 
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the execution. If init; precedes init;, then Lemma 5.1 implies the result, so assume that init; 
precedes init;. 

Since a round 2 message from 7 is in the Net, a round 1 message (j,1,*,W’) is also. Then 
Lemma 5.1 implies that 7 € W’. 

We claim that j does not leave or fail before the init;. Suppose for the sake of contradiction 
that it does. Then the net_mcast(j,2,*,*) event precedes the init;. Then environment well- 
formedness implies that 7 does not fail or leave prior to the net_mcast(j,2,*,*) event, because it 
initiates after this event. Also, i does not abstain, because it initiates. And 7 does not decide prior 
to the net_mcast(j,2,*,*) event, because that precedes the init;. Therefore, 1 ¢ failed[1] ;U 
out [1]; in the pre-state of the net_mcast(j,2,*,*) event, soi € world[1,j]l; \ failed[1],U 
out [1]; in that state. The precondition of net_mcast implies that value[1,i]; # 1 in the pre- 
state, that is, 7 has received a round 1 message from 7 before the net_mcast(j,2,*,*). But this 
cannot happen, because init; happens after the net_mcast(j,2,*,*). This contradiction implies 
that 7 does not leave or fail before the init;. Then world consistency implies that 7 € W, as needed. 

|_| 


In the rest of this section, we consider a situation where no failures happen from some point 
onward in an execution, and where the rounds of all processes are at most r at the point where 
failures cease. The following lemma says that all round r+2 messages that are ever sent have the 
same world component. 


Lemma 5.2 Suppose thatr > 0. Suppose that there is a point t in an execution such that every 
process has round < r at point t, and no fail events happen from t onward. 

If net_mcast(i,r+2,v,W) and net_mcast(j,r+2,v’,W’) both occur in the execution, then W = 
W?. 


Proof: We show that W C W’. The other direction is analogous. 

The two sets are determined in the precondition of net_mcast, as follows: 

W = world[r+1,i]; \ out[r+1]; \ failed[r+1];, where the values of the last two terms are 
taken from the pre-state of net_mcast (i,r+2,v,W), and 

W? = world[r+1,j]; \ out[r+1]; \ failed[r+1],, where the values of the last two terms are 
taken from the pre-state of net_mcast(j,r+2,v’,W’). Invariant A.7 implies that W = world[1,il]; 
\ out-by[r+1]; \ failed-by[r+1],;, where the values of the last two terms are taken from the 
pre-state of net_mcast (i,rt+2,v,W), and 

W? = world[1,j]; \ out-by[rt+1]; \ failed-by[r+1];, where the values of the last two terms 
are taken from the pre-state of net_mcast(j,r+2,v’,W’). 

Consider some k € W. The precondition of net_mcast(i,r+2,v,W) implies that in the pre- 
state, value[rt1,k]; # 1, that is, 7 has received a round r+1 message from k. This means 
that k has previously sent a round r+1 message. Since (by assumption) r > 0, it follows that r 
+1 > 2, which means that k has sent a round 2 message. Invariant 5.1, applied to any state after 
both net_mcast(j,1,*,*) and net_mcast(k,2,*,*) have occurred, implies that k is in the world 
component of j’s round 1 message, and so k is put into world[1,j]; when that is defined. To 
prove that k € W’, it suffices to show that k is never placed into either of the sets out-by[r+1] ; or 
failed-by [r+1] ;. 

First, we show that k is never placed into out-by[r+1];. Suppose for the sake of contradiction 
that k is put into out-by[r+1]; at some point in the execution. Then consider some state that 
occurs after this has happened, and that is not before the pre-state of net_mcast(i,r+2,v,W). In 
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this state, we have both value [r+1,k]; # 1 andk € out-by[r+1],. This contradicts Invariant A.4. 
Therefore, k is never placed into out-by[r+1] ;. 

Second, we show that k is never placed into failed-by[r+1];. Suppose for the sake of con- 
tradiction that k is put into failed-by[r+1]; at some point in the execution. Then k fails in the 
execution, which implies that it fails before point ¢t. But we have already noted that k sends a 
round r +1 message during the execution. It does not send this before point t, because that would 
mean that it would reach round r +1 before point t, contradiction our assumptions. So k sends the 
round r +1 message after point ¢, and so it cannot fail before point t, a contradiction. Therefore, 
k is never placed into failed-by [r+1] ;. | 


The next lemma says that, under the same assumptions as for the previous lemma, all the round 
r+2 messages have the same value component. 


Lemma 5.3 Suppose thatr > 0. Suppose that there is a point t in an execution such that every 
process has round < r at point t, and no fail events happen from t onward. 

If net_mcast(i,r+2,v,W) and net_mcast(j,r+2,v’,W’) both occur in the execution, then v = 
Vv’. 


Proof: Process i determines v as the minimum of all values value [r+1,k]; for all k € W, and 
process i determines v’ as the minimum of all values value [rt+1,k]; for all k € W’. Lemma 5.2 
implies that W = W’. Since values are consistent (by Invariant A.2), the sets of values over which 
the two minima are taken are identical. Therefore, v = v’. | 


Finally, we prove the main early-deciding theorem. It says that, if no failures happen from 
some point onward and the rounds of all processes are at most r when failures cease, then no CUP 
participant ever advances beyond round r +2. Since we have already proved termination, this 
implies that all active CUP participants decide by round r +2. 


Theorem 5.4 Suppose thatr > 0. Suppose that there is a point t in the execution such that every 
process has round < r at point t, and no fail events happen from t onward. 
Then every process always has round < r +2. 


Proof: Lemmas 5.2 and 5.3 yield a common value and world for round r+2 messages. Fix v’ and 
W’ to be the common value and world, respectively. 

We show that the precondition of net_mcast(i,r+3,*,*) can never be true, which implies 
that such an event can never happen. This implies that every process always has round < r +2. 
Suppose for the sake of contradiction that the precondition of net_mcast (i,r+3,v,W) is true in 
some reachable state s, for some fixed 2. 

Since the precondition holds in s, world[r+2,i]; # . in s, and so Invariant A.1 implies 
that some (i,r+2,v’’,W’’) message is in the Net in s, where v’? = value[r+2,i]; and W’’ = 
world[r+2,i];. Since v’ and W’ are the common value and world for round r+2 messages, this 
implies that value [r+2,i]; = v’ and world[r+2,i],; = W’. 

We show that for all 7 € world[r+2,i]; \ out[r+2];, value [r+2,j]; = value[r+2,i]; and 
world[r+2]; C world[r+2,i];. This suffices to show that the final precondition fails, which yields 
a contradiction. 

Fix j € world[r+2,i]; \ out [r+2],. Since failed[r+2]; = { }, iffollows that 7 € world[r+2,i]; 
\ out[r+2]; \ failed[r+2];. The precondition of the net_mcast then implies that value [r+2,j]; 4 
t in state s. Invariant A.1 then implies that some (i,r+2,v’’?’,W’’?’) message is in the Net in s, 
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where v’’?? = value[r+2,j];andW’’’? = world[r+2,i];. Since v’ and W’ are the only value and 
world for round 2 messages, this implies that value [r+2,j]; = v’ and world[r+2,j]; = W’ in 
state s. Thus, value[r+2,j]; = value[r+2,i]; and world[r+2]; C world[r+2,i];, as needed. 

|_| 


Note that this proof does not work for the case where r=0, because of potential differences in 
the initial worlds of correct processes. Consider, for example, an execution in which no process 
ever fails, and some process, k, leaves after sending a round 1 message. Process k may be included 
in the initial world of process 7 but not in the initial world of another process 7, if 7 initiates CUP 
after k leaves. In this case, i takes k’s round 1 message into account when choosing its round 2 
message, while 7 does not (because & is not in j’s initial world). This scenario can only occur in 
round 1, because no process can send a round 2 message before j initiates. 

For the case where r = 0, the best we can state is: 


Corollary 5.5 Suppose there is a point t in the execution such that every process has round = 0 
at point t, and no fail events happen from t onward. 
Then every process always has round < 3. 


Proof: This is immediate from Theorem 5.4, using r = 1. | 


5.4 Timing Assumptions 


For the sake of analyzing the performance of the CUP algorithm, we use timed I/O automata [21, 
Ch. 23]. We can regard an ordinary I/O automaton as a special case of the timed model, in which 
arbitrary amounts of time can pass between events. All the safety results carry over to this model. 

For this analysis, we add an extra assumption: we assume that any action that is enabled either 
gets performed or gets disabled by another action, before any time passes. 


5.5 Latency Analysis 


We now analyze the algorithm’s latency in executions in which there are time bounds on certain 
environment actions. We assume the following bounds: 


1. 6, is an upper bound on message latency. That is, if a netrcv(m) event occurs, the time 
since the corresponding net_mcast (m) is at most 61. 


2. dg is an upper bound on failure and leave detection time. Moreover, if a message is lost due to 
failure, then the failure is detected at most 62 after the lost message was sent. More precisely, 


(a) Assume init;(*,W) occurs with 7 € W and fail; or leave; occurs at time t. Then 
fail_detect;(j), leave_detect;(j), decide;, leave;, or fail; occurs by time ¢ + do. 


(b) Define U = Uger{W| init,(*,W) occurs}. Assume i,j € U and net_mcast,;(m) oc- 
curs at time ¢ but no net_rcv;(m) occurs. Then fail_detect;(j), leave_detect,;(j), 
decide,;, leave;, or fail; occurs by time ¢ + do. 


3. 63 is an upper bound on the time difference between the initiation time of different processes. 
More precisely: 
Assume some process initiates at time ¢ and does not fail by time t+ 6;. Assume further that 
init;(*, W) occurs. Then, every process j7 € W initiates, abstains, leaves, or fails by time 
t+ 63. 
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In practice, the failure detection time would be at least as large as the message latency. We 
therefore assume that 69 > 64. 

We now use the above bounds on the environment to establish bounds on CUP’s running times. 
The next lemma bounds the time it takes from when some process initiates CUP until all processes 
terminate round 1. 


Lemma 5.6 Assume that some process initiates CUP at time t and does not fail by time t + 61. 
Then by time t+ 62+ 63, every process that initiates either terminates round 1, or leaves, or fails. 


Proof: Let i be a process that initiates and does not leave or fail by time t + d2 + 63. We now 
show that 2 terminates round 1 by time ¢ + d2 + 63. If% decides by time t+ 62 + 63, then we are 
done. We therefore assume that 7 does not decide by this time. 

In order to terminate round 1, 7 has to have a round 1 message from every process j7 € 
world[1,i]; \ out[1]; \ failed[1];. That is, for every process 7 € world[1,i]j;, ¢ has to re- 
ceive a round 1 message or an OUT message from j, or a fail_detect;(j) or a leave_detect;(j) 
event. 

Fix a process 7 Eworld[1,i]j, i-e., 7 is in 2’s initial world. Since some process initiates at time 
t, by our assumption on initiation times, j initiates, abstains, leaves, or fails by time ¢ + 43. 

If 7 fails or leaves by time ¢ + 63, then by our assumption on failure and leave detection times, 
fail_detect;(j) or leave_detect;(j) occurs by time t + dp + 63 (since we assume that 7 does not 
decide, leave, or fail by this time), and we are done. 

Assume now that j does not fail or leave by time t + 463. Since 7 is in 2’s initial world, j either 
initiates or abstains by this time, at which point 7 sends a round 1 message or an OUT message 
(resp.). If 2 receives this message, 7 receives it by time t+ 63+ 6. If 7 does not receive this message, 
fail_detect(j); or leave_detect(j),; occurs by time t + 63 + 69. 

Since dg > 61, we get that for every 7 Eworld[1,i];, by time ¢ + 63 + d9, 7 either receives a 
round 1 message or an OUT message from j or a fail_detect(j); or leave_detect(j); event 
occurs. a 


The following lemma bounds the duration of subsequent rounds. 


Lemma 5.7 Assume that by time t, every process that initiates CUP either terminates round r 
> 0, or decides, or leaves, or fails. Then, by time t+ d2, every process that initiates CUP either 
terminates round r+1, or decides, or leaves, or fails. 


Proof: Consider a process i that initiates CUP and does not leave or fail or decide by time t+ do. 
We now show that 7 terminates round r+1 by time t + do. 

In order to terminate round r+1, 7 has to have a round r+1 message from every process j € 
world[r+1,i]; \ out[rt+1]; \ failed[r+1];. That is, for every process 7 € world[rt+1,i]j, i 
has to either receive a round r+1 message or an OUT message from j, or a fail_detect,;(j) or a 
leave_detect;(j) event has to occur. 

Fix a process 7 €world[r+1,i];. Process j7 must have initiated. By time t, j7 terminates 
round r+1, or decides, or leaves, or fails. If 7 leaves or fails by time t, then fail_detect (j); or 
leave_detect(j); occurs by time t+ dg. Otherwise, 7 sends a round r+1 message or an OUT 
message (in case it decides) by time t. If i receives this message, i receives it by time t + dj. 
Otherwise, fail_detect (j); or leave_detect (j); occurs by time t+ dg. Since dg > 61, we get that 
z terminates round r+1 by time t + 69. | 
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Using the two lemmas above, we get the following bound on the running time of an execution 
of CUP with r rounds. 


Lemma 5.8 Assume that some process initiates CUP at time t and does not fail by time t + 61. 
Ifi decides at round r > 0, it does so by time t+ 63 + rq. 


Proof: By Lemma 5.6, by time t + 63 + 62, every process that initiates CUP either terminates 
round 1, or leaves, or fails. By iterative application of Lemma 5.7, we get that by time t+ 63 + d2+ 
(xr — 1)d9 = t+ 63 + rg, every process that initiates CUP either terminates round r, or decides, or 
leaves, or fails. | 


As a consequence of the above lemmas and the early-deciding theorem of the previous section 
we get the following theorem: 


Theorem 5.9 Suppose that there is a point t in the execution such that no fail events happen 
from t onward. Suppose also that some process initiates CUP by time t. Then every process that 
decides, decides by time t + 63 + 369. 


Proof: Let r be the highest value of round of any process at time t. Since some process initiated 
CUP by time t, r > 0. By Theorem 5.4, every process that decides, decides at the end of round 
r+2 at the latest. 

We consider two cases. First, if r > 1, then by Invariant A.12, every process that initiated 
CUP has either terminated round r-1 or left or failed by time t. By applying Lemma 5.7 three 
times, we get that every process that initiates CUP either terminates round r+2 or leaves or fails 
by time ¢ + 369. Therefore, in this case, every process that decides, decides by time t + 3d. 

Next, assume that r = 1. Since some process initiates CUP by time ¢ and does not fail, by 
Lemma 5.6, by time ¢ + 63 + 62, every process that initiates CUP either terminates round 1, or 
leaves, or fails. By applying Lemma 5.7 twice, we get that every process that initiates CUP either 
terminates round r+2 or leaves or fails by time ¢ + 63 + 369. Therefore, in this case, every process 
that decides, decides by time ¢ + 43 + 360. | 


6 Environment and Model Assumptions for Atom 


6.1 Timing Assumptions 


We model time using a continuous global variable now, which holds the real time. This is a real 
variable, initially 0. We assume that it increases with derivative 1. Each endpoint 7 is equipped 
with a local clock, clock;, modeled by a continuous, bijective, monotonically increasing function 
from the nonnegative R to the nonnegative R. 

We assume a bound of [ on clock skew, where IT is a positive real number. Specifically, for 
each endpoint i, we assume that in any state of the system that is reachable |clock; — now| <T'/2. 
That is, the difference between each local clock and the real time is at most I'/2. It follows that 
the clock skew between any pair of processes is I’, formally: in any reachable state, and for any two 
endpoints i and j, |clock; — clock;| <T. 

We assume that local processing time is 0 and that actions are scheduled immediately when 
they are enabled. Formally, when any locally controlled action of any process that is part of our 
local algorithm is enabled, then before any time passes, the action is either performed or becomes 
disabled. 
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6.2 Reliable Network Assumptions 


We assume that we are given a low-level reliable network service Net. Like DAB, Net is parame- 
terized by a message alphabet, M. 

The Net(M) signature is defined in Figure 6. The actions are the same as those of DAB, except 
that they are prefixed with net_. 


Input: 


net_join;, net_leave;, fail,;, ie€I, 
net_mcast,(m), meM, icl 


Output: 


net_join_OK;, net_leave_OK;, iclI, 


net_rcv,(m), meM, icl 


Figure 6: The signature of the Net service. 


Net(M) assumes that its application satisfies the same basic safety conditions as those specified 
above for DAB(M), except that action names are preceded with net_. Assuming the application 
satisfies these conditions, Net(/) satisfies a number of safety and liveness properties. 

First, Net satisfies the basic properties specified above for DAB: join/leave integrity, message 
integrity, eventual join, and eventual leave. All of these properties are the same as for DAB, except 
that action names are prefixed with net_. 

In addition, Net guarantees FIFO delivery of messages: 


e FIFO delivery: If net_mcast;(m) occurs before net_mcast;(m’), and net_rcv;(m’) occurs, 
then net_rcv;(m) occurs before net_rcv;(m’). 


Net(M/) also satisfies the following liveness property: 


e Eventual delivery: Suppose net_mcast;(m) occurs after net_join_OK;, and no fail; occurs. 
Then either net_leave; or fail; or net_rcv;(m) occurs. 


Additionally, the network latency is bounded by a constant nonnegative real number A. For- 
mally, Net(M) guarantees: 


e Message latency: If net_rcv;(m) occurs, then the real time elapsed since the corresponding 
net_mcast;(m) is at most A. 


The maximum message latency of A guaranteed by Net is intended to include any pre-send 
delay at the network module of the sending process. 

Since an implementation of Net cannot predict the future, it must deliver messages within time 
A as long as no failures occur. In particular, if a message is sent more than A time before its sender 
fails, it must be delivered. 


7 The Atom Algorithm 


The Atom algorithm consists of a collection of processes corresponding to the different endpoints 
in J. It uses Net and CUP services as building blocks. It uses multiple instances of CUP. 
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7.1 Data Types 


Atom defines the constant ©, a positive real number. This will represent a time slot. We assume 
that O > A. 

Recall that M represents the message alphabet of DAB. We will use M’' to represent the message 
alphabet of Net. We define the message alphabet of Net in term of the alphabet of Atom: 


e M,, the set of finite sequences of elements of M. These are the bulk messages processes send. 
e M, = M, U{JOIN, LEAVE} U{CUP — INIT} x I} 
e M'=Ix MoxN. 


M’ is the complete message alphabet of Net. Each message contains either a bulk message (sequence 
of client messages) for a particular slot, a request to join or leave a particular slot, or a report that 
process has initiated consensus on behalf of a particular endpoint. Each message is tagged with 
the sender and the slot. 


7.2 Using the Net and CUP 


The Net service alphabet is instantiated with M’. That is, Atom uses a service Net(M]’) to im- 
plement the service DAB(M). Atom uses multiple instances of CUP, at most one for each process 
j. 

As before, a fail; action causes process i to stop. fail; actions go to all the components, i.e., 
Net and all instances of CUP (including dormant ones), and cause all of them to stop taking any 
locally controlled actions. Since fail; actions cannot be intercepted, we do not include them in 
the code. 

leave; actions also go directly to all the local instances of CUP, including dormant ones. 


7.3 Atom Algorithm Overview 


The algorithm divides time, and respectively, messages, into slots. As time advances, each process 
advances through slot. The duration of a slot is O. 

Each process multicasts all of its messages for a given slot in one bulk message. This is a 
useful abstraction that we make in order to simplify the presentation and analysis of the Atom 
algorithm. In practice, the bulk message does not have to be sent as one message; a standard 
packet assembly /disassembly layer can be used to provide all-or-nothing behavior. 

Message delivery is also done in order of slots. Before delivering messages of a certain slot s, 
each process has to determine the membership of this slot, that is, the set of processes from which 
to deliver messages in this slot. To ensure total order, all the processes that deliver messages for a 
certain slot have to agree upon the membership of each slot. For each slot, messages are delivered in 
the order of process indices, and for each process, the messages are unpacked from its bulk message 
and delivered in FIFO order. 


7.4 Signature 


The signature of Atom at process 7, Atom,, is presented in Figure 7. It includes all the interaction 
with the client and all the interaction with the underlying network. The implementation of Atom 
uses CUP as a building block. Hence Atom; has additional input and output actions for interacting 
with CUP. Since Atom uses multiple instances of CUP, at most one for each process 7, actions of 
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CUP automata are prefixed with CUP(j). For example, process 7 uses the action CUP(j).init; to 
initiate the CUP automaton associated with process 7. CUP.fail and CUP.leave are not output 
actions of Atom, since they are routed directly from the environment to all instances of CUP. 

The signature of Atom; also includes two internal actions, end_slot, and members. These two 
actions play a role in determining the membership for each slot. end_slot (s); occurs at a time by 
which slot s messages from all processes should have reached process 7. At this point, processes 
from which messages are expected but do not arrive are suspected to have failed. For each suspected 
process j, CUP(j) is run to have the surviving processes agree upon j’s failure slot. This is needed 
because failed processes can be suspected at different slots by different surviving processes. After 
CUP reaches decisions about all the suspected processes that could have failed at slot s, members (P, 
s) can occur, with P being the agreed membership for slot s. When process i performs members (P, 
s);, all the messages included in bulk messages that 7 received for slot s from processes in P are 
delivered (their delivery is triggered) in order of process indices. 


Input: 


join,;, leave, 
net_join_OK;, net_leave_OK; 
mcast, (m), meM 
net_rcv,(m), meM’ 

fail; 

CUP (j) .decide,;(v), veN 


Output: 


join_OK;, leave_OK; 


net_join,;, net_leave, 
net_mcast,; (i, m, s), m€My, seN 
rev, (m), meM 

CUP(j).init;(v, W), veN 


CUP(j) .abstain; 


CUP (j) .leave_detect,(j), jel 
CUP (j) .faildetect;(j), jel 


Internal: 


end_slot,;(s), seN 
members; (P, s), P set of I, sd 


Figure 7: Atom;: Signature. 


7.5  Pseudo-code 


The Atom; code is presented in Figures 8-10. The state components are presented in Figure 8. 

Recall that we do not assume that processes execute the algorithm from the beginning of 
time. Rather, the application issues an explicit join event, and waits for a join_OK. The variable 
join-slot holds the slot at which a process starts participating in the algorithm; this will be the 
value of current-slot when join_OK will be issued, and the first slot for which a bulk message 
will be sent. If a process explicitly leaves the algorithm, its leave-slot holds the slot immediately 
following the last slot in which the process sends a bulk message. Both join-slot and leave-slot 
are initially oo, so as to be larger than any actual slot number they are compared with. 
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clockeR, initiallyel0, [/2]; dynamic type: continuous functions 
join-slot € NU o, initially oo 

leave-slot € NU oo, initially oo 

did-join-OK, boolean, initially false 

did-leave, boolean, initially false 

mcast-slots CN, initially {} 

ended-slots CN, initially {} 

reported-slots CN, initially {} 


for every s € N 
out-buf[s] € Mj, initially empty sequence of M 
joiners[s] C I, initially {} 
leavers[s] C I, initially {} 
suspects[s] CI, initially {} 
for every sE€N, j € I 
in-buf[j,s], j € I, s € N, finite sequence of M or Ll, initially L 


for every j € I\{ i } 
CUP-status[j] € { idle, req, running, done }, initially idle 
cUP-req-val[j] €NUf{t1}, initially L 
cUP-dec-val[j] €eNU{ 1}, initially 1 


derived variables: 
current-slot € N= | clock / O | 


for every s € N 
alive[s] CI={j | in-buf[s,j] #1} 


Figure 8: Atom,: State. 


The boolean flags did-join-OK and did-leave are used to ensure that join_OK and net_leave 
actions will not be performed more than once. The set mcast-slots keeps track of the slots for 
which the process already multicast a message (JOIN, LEAVE, or bulk). Likewise, ended-slots 
and reported-slots keep track of the slots for which the process already performed the end_slot 
or members actions, resp. 

out-buf [s] stores the message (bulk, JOIN, or LEAVE) that is multicast for slot s; it initially 
holds an empty sequence, and in an active slot, all application messages are appended into it. 
A JOIN message is inserted for the slot before the join-slot, and a LEAVE message for the 
leave-slot. Either way, there is no overlap with a bulk message. 

The variables joiners[s] and leavers[s] keep track of the processes j for which join-slot; 
=s (resp. leave-slot; =s). suspects[s] is the set of processes suspected in slot s as determined 
when end_slot(s) occurs. 

The variable in-buf[j,s] is a finite sequence of messages received in a slot s bulk message 
from process j7. The data type finite sequence supports assignment, extraction of the head of the 
queue, and testing for emptiness. 

There are three variables for tracking the status and values of the different instances of CUP. 
CUP-status[j] is initially idle; when CUP(j) is initiated, it becomes running; if a CUP-INIT 
message for j arrives, it becomes req; and when there is a decision for CUP(j), or if the process 
abstains from CUP(j), it becomes done. CUP-req-val[j] holds the lowest slot value associated 
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with a CUP-INIT message for 7 (-L if no such message has arrived). Finally, CUP-dec-val[j] holds 
the decision reached by CUP(j), and 1 if there is none. 
alivel[s] is a derived variable, storing the set of processes from which slot s bulk messages 
were received. 
join, 
Eff: trigger (net_join,) 
net_join_OK,; 
Eff: join-slot ¢ current-slot + 2+ [ T/0 ] 
out-buf[join-slot - 1] « JOIN 
join_OK; 
Pre: did-join-OK = false 
current-slot = join-slot 
Eff: did-join-OK < true 
leave; 
Eff: if (join-slot € N) then 
leave-slot ¢ max(current-slot, join-slot) + 1 
out-buf [leave-slot] « LEAVE 


net_leave,; 
Pre: did-leave = false 
leave-slot € mcast-slots 


Eff: did-leave ¢ true 


net_leave_OK 
Eff: trigger (leave_OK; ) 


mcast ; (m) 
Eff: if (join-slot < current-slot < leave-slot) then 
append m to out-buf [current-slot] 


net_mcast, (i, m, s) 
Pre: join-slot € N 
join-slot - 1<s < leave-slot 
current-slot = sti 
s ¢ mcast-slots 
m = out-buf [s] 
Eff: mcast-slots ¢ mcast-slots Uf{s } 


net_rcv,(j, JOIN, s) 
Eff: joiners[st+i] ¢ joiners[st+i] U { j } 


net_rcv,(j, LEAVE, s) 
Eff: leavers[s] ¢+ leavers[s] U { j } 
foreach (k such that CUP-status[k] = running) do 
trigger (CUP (k) .leavedetect, (j)) 


net_rcv,(j, m, s), m sequence of M 
Eff: in-buf[j,s] «+ m 


Figure 9: Atom;: Transitions related to multicast, join, and leave. 
In Figure 9 we present the first part of Atom’s transitions, including transitions related to 


joining, leaving, multicasting messages, and receiving messages from the network. ‘Transitions 
related to membership and totally ordered delivery are presented in Figure 10. 
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When the application issues a join, Atom triggers net_join. Once the Net responds with a 
net_join_OK, Atom calculates the join-slot to be 2+ [I/O] slots in the future. This will allow 
enough time for the join message to reach the other processes. A JOIN message is then inserted 
into out-buf [join-slot-1]. Once the current-slot reaches join-slot, join_OK is issued to the 
application. 

When the application issues a leave, the leave-slot is chosen to be the ensuing slot, and a 
LEAVE message is inserted into out-buf [leave-slot]. A net_leave is issued after the LEAVE 
message has been multicast, and the net_leave_OK triggers a leave_OK to the application. 

Messages multicast by the application are appended to the bulk message for the current slot 
in out-buf [current-slot]. Once a slot s ends, the message pertaining to this slot is multicast 
to the other processes using net_mcast. If s = join-slot - 1, a JOIN message is sent. If s = 
leave-slot, a LEAVE message is sent, and if s is between join-slot and leave-slot - 1, a 
bulk message is sent. 

When a bulk message is received, it is stored in the appropriate in-buf. When a JOIN (LEAVE) 
message is received, the sender is added to the joiners (resp. leavers) set for the appropriate slot. 
Additionally, when a LEAVE message is received, CUP. leave_detect is triggered for all running 
instances of CUP. 

Process 7 performs end_slot;(s) once it should have received all the slot s messages sent by 
other non-failed processes. Since slot s messages are sent immediately when slot s ends, messages 
are delayed at most A time in Net, and the clock difference is at most IT, process 2 should have 
all the non-failed processes’ slot s messages A+T time after slot st1 began. At this time, clock 
> (s+1)0+A+T. Process i expects to receive slot s bulk messages from all the processes that 
are in alive[s-1], except for those that are leaving in slot s. Any process from which a slot s 
bulk message is expected but does not arrive becomes suspected at this point, and is included in 
suspects[s]. 

For every suspected process, CUP is run in order to agree upon the slot at which the process 
failed. The slot s in which the process is suspected is used as the initial value for CUP. The estimated 
world for CUP is alive[s] U joiners[st1]. This way, if & joins in slot s+1, k is included in 
the estimated world. This is needed in order to satisfy the world consistency assumption of CUP, 
because k can detect the same failure at slot s+1, and therefore participate in CUP(j). When 7 
initiates CUP(j), it also multicasts a (CUP-INIT, j) message. If a process k does not detect the 
failure and does not participate, the (CUP-INIT, j) message forces k to abstain. 

Since Atom implements the failure detector for CUP, the effect of end_slot;(s) also triggers 
CUP (k).fail_detect(j) actions for every suspected process 7, and for every currently running 
instance k of CUP. 

Process 7 abstains from CUP(j) only if a (CUP-INIT,7) message has previously arrived, setting 
CUP-status[j]; = req, and only if end_slot; has already occurred for a slot value greater than 
CUP-req-val[j];. The latter condition ensures that 7 abstains only from instances of CUP that it 
will not initiate. This is because the network guarantees that when a process fails, at most one slot 
bulk message from this process is lost (since we assume that Delta < 0). This implies that the 
detection of 7’s failure by two non-failed processes can occur at most one slot apart. Therefore, if 
end_slot; has already occurred for a slot value greater than CUP-req-val[j];, 2 will never suspect 
j. 

The members(P, s) action triggers the delivery of all slot s messages from processes in P. It can 
only occur once agreement has been reached about the processes to be included in P. Since the slot 
at which a process k is suspected by two processes 7 and 7 can differ by at most one, members, (P, 
s) can occur after 7 receives decision from all instances of CUP pertaining to processes suspected in 
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end_slot, (s) 
Pre: join-slot <s 
leave-slot = co 
s ¢ ended-slots 
clock ) (st1)O+ A +T 
Eff: ended-slots ¢ ended-slots U{s } 
suspects[s] ¢ (alive[s-1] U joiners[s] \ leavers[s]) \ alive[s] 
foreach (j € suspects[s]) do 
trigger (CUP(j).init,;(s, alive[s] U joiners[s+1])) 
net_mcast, (i, (CUP-INIT, j), s) 
CUP-status[j] «+ running 
foreach (k such that CUP-status[k] = running) do 
trigger (CUP (k) .fail_detect,(j)) 


net_rcv,(j, (CUP-INIT, k), s) 

Eff: if (CUP-status[k] = idle V CUP-req-val[k] ) s) then 
CUP-status[k] + req 
CUP-req-val[k] ¢ s 


CUP (j) .abstain; 
Pre: CUP-status[j] = req 
ds € ended-slots : s > CUP-req-val[j] 
Eff: CUP-status[j] < done 
CUP (j) .decide, (s) 
Eff: CUP-status[j] < done 
CUP-dec-val[j] ¢+ s 
members, (P, s) 
Pre: s = min{ ended-slots \ reported-slots } 
s + 1 € ended-slots 
Vj € (suspects[s] U suspects[st+1i]) : CUP-status[j] = done 
P={ j € alive[s] | CUP-dec-val[j] =1 V CUP-dec-val[j] >s } 
Eff: reported-slots + reported-slots U { s } 
foreach j € P, in order of indices do 
while in-buf[j,s] not empty do 
trigger (rcv, (head (in-buf [i,s]))) 


Figure 10: Atom;: Transitions related to membership and message delivery. 


slots up to s+1. Therefore, members;(P, s) must occurs after end_slot (s+1), when the suspicions 
for slot sti are determined. The set P includes every process 7 that is alive in slot s and for which 
there is either no CUP instance running (in which case j was not suspected), or the CUP decision 
value is greater than s. 


7.6 Latency Analysis 


In this section we analyze the latency guarantees of Atom. In Section 7.6.1 we show that in failure 
free executions, Atom’s message latency is bounded by A + 20 + 21. We denote this bound by 
A atom. In Section 7.6.2, we assign values to the constants that were used in the analysis of CUP 
in Section 5.5 (6), 62, and 63). Then, in Section 7.6.3, we consider executions in which failures do 
occur but there is a long time period with no failures. We analyze the time it takes Atom to clear 
the backlog it has due to past failures, and reach a situation in which message latency is bounded 
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by the same bound as in failure free executions, namely A 4iom, barring additional failures. 

The fact that once failures stop for a bounded time all messages are delivered within constant 
time implies that in periods with f failure, Atom’s latency is at most linear in the number of failing 
processes. 


7.6.1 Failure free executions 


Lemma 7.1 The time from when process j starts slot s (i.e., current-slot; becomes s) until 
process 1 performs end_slot;(s+1) is at most A+ 20 + 27. 


Proof: According to its preconditions, end_slot;(s) occurs 20 + A+T time after 7 starts slot 
s. Since the difference between two processes’ clocks is at most I’, 7 starts slot s at most I’ time 
after 7 starts this slot. | 


Lemma 7.2 Consider an execution in which no process fails. If the application at process 7 per- 
forms mcast;(m) when current-slot; = s and if process i delivers m, then i delivers m immedi- 
ately after end_slot;(s+1) occurs. 


Proof: If delivers m, rcv;(m) is triggered during the Members;(P,s) action. Since no process 
fails, suspects[s];U suspects[s+1]; is an empty set, and thus the only precondition that needs 
to be satisfied in order to perform Members; (P,s) is st1 € ended-slots,, which is true immediately 
after end_slot;(st+1) occurs. | 


As a direct result of these two lemmas, we get the following theorem: 


Theorem 7.3 If the application at process 7 performs mcast;(m) at time t, and if process i delivers 
m, thenz delivers m by time t+ Ayton =t+ A+ 20 4 QI. 


7.6.2 CUP bounds 


We now assign values to the constants used in the analysis of CUP in Section 5.5. Recall, 6; is an 
upper bound on message latency; 62 is an upper bound on failure and leave detection time, and if 
a message is lost due to failure, then the failure is detected at most 62 after the lost message was 
sent; and 63 is an upper bound on the difference between different processes’ initiation times. 


Lemma 7.4 6, =A 


Proof: By definition, both A and 6, are defined to be upper bounds on the underlying network 
latency. | 


Lemma 7.5 69 = A+304 272 


Proof: Assume that CUP(k).init;(*,W) occurs with 7 € W. Assume that one of the follow- 
ing happens at time ¢: fail;, leave;, or net_mcast;(m) for a message m that is lost because j 
subsequently fails. Let s be the value of current-slot; at time t. Assume also that by time 
t+A+30+4 21, ¢ does not decide, leave, or fail, so CUP-status[k]; = running and i is active at 
this time. We have to show that by this time, fail_detect;(j) or leave_detect;(j) occurs. 

If j fails at time t, then 7’s slot s message is never sent, and therefore z detects the failure and 
invokes CUP(k) .fail_detect;(j) during end_slot;(s) at the latest. By Lemma 7.1, this occurs 
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by time t+ 20 + A+ 2P. Likewise, if 7 sends a message m while current-slot; = s, and m is 
lost, then by the FIFO nature of the network, j’s slot s message is also lost and 7 detects j’s failure 
during end_slot;(s) at the latest. 
Assume next that 7 leaves when current-slot; = s, i.e., j’s leave-slot is st+1. If 7 receives 
a LEAVE message from 7, it receives it before end_slot;(s+1) occurs, and immediately triggers 
CUP (k).leave_detect;(j). Otherwise, 7 receives no slot s+1 message from j and suspects 7 and 
invokes CUP(k) .fail_detect;(j) during end_slot;(st1). This occurs by time t+ 30+ A+2I. 
a 


Lemma 7.6 63 =I'+0 


Proof: Assume that some process process | initiates CUP(k) at time ¢ and does not fail by time 
t+ A. Assume further that CUP(k).init;(*, W) occurs with 7 © W. We have to show that j 
initiates, abstains, leaves, or fails by time t +I + O. 

Process [| triggers CUP(k).init;(s, *) during the end_slot;(s) action, and k Csuspects,[s]. 
If j initiates CUP(k), there is a slot s’ such that j triggers CUP (k) .init; during the end_slot ;(s’) 
action, and k €suspects;[s’]. By Invariant A.19, s’ < s+1. Therefore, CUP(k).init,; occurs 
no later than time t+ I+ ©, and we are done. 

Assume now that j does not initiate CUP(k), and does not leave or fail by time t +I +0. We 
now show that j abstains from CUP(k) by time #+T+ 0. 

When CUP(k).init;(s, *) is triggered, | multicasts a (CUP-INIT, k) message. By Lemma A.8, 
net_join_OK; occurs before | initiates CUP(k), that is, before / multicasts this message. Moreover, 
by assumption, J does not fail by time t+ A and 7 does not leave or fail by time t + A (because 
A <0). Therefore, 7 receives this message by time t + A, which is before time t+I+ 0. After j 
receives this message, CUP-status[k]; is req and CUP-req-val [k] ; is less than or equal to s. By 
time t+ I+ ©, end_slot;(s+1) occurs and the condition for CUP(k).abstain,; becomes true, and 
remains true until CUP(k).abstain,; occurs and changes CUP-status[k] ;. Therefore, before any 
time passes, CUP(k) .abstain,; occurs. | 


7.6.3 Failure free periods 


We now consider executions in which failures do occur but there are long time periods with no 
failures. We analyze the time it takes Atom to clear the backlog it has due to past failures, and 
again reach a situation in which message latency is bounded by A ygom, barring additional failures. 

Let t; = 63 + 469, where 63 and dj are bounds as given above for the difference between 
process initiation times and failure detection time, resp. From Lemmas 7.6 and 7.5 we get that 
ty =T4+04+4(A+4 304 27) = 4A+ 9F 4+ 130. 

Assume that from time t to time t’ = t+ t, there are no failures. We now show that if a message 
m is sent after time t’, and there are no failures for a period of length A jim after m is sent, then m 
is delivered within A 4rgm time of when it is sent. Since the delivery order preserves the FIFO order, 
this also implies that any message m’ sent before time t’ is delivered by time ¢’ barring failures in 
the Aytom time interval after m’ is sent. 


Theorem 7.7 Assume no process fails between time t and =t+t,. If mcast(m); occurs at a 
time t" such that t+t, < t", and no failures occur from time t” to time t" + Atom, and ifi delivers 
m, then i delivers m by time t" + Aatom. 
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Proof: By Lemma 7.5, by time t+ 62 all the processes detect all the failures that occur by time t. 
Therefore, no process initiates an instance of CUP after time t+65. Since no failures occur after time 
t+62, by Theorem 5.9, all CUP instances that i initiates terminate by time t+ 69+ 63+ 362 = t+11. 

Let s be the value of current-slot, at time t” (i.e., when mcast (m); occurs). By Lemma 7.1, 
process 7 performs end_slot;(s+1) by time t”’ + A+20+4 2P = t”+ Atom. At this time, there are 
no active CUP instances, because CUP instances pertaining to failures that occurred before time t¢ 
have all terminated and no new failures occur until time t” + Atom. Therefore, for every slot s’ 
<_s, in order of slot numbers, Members(P, s’); becomes enabled until it occurs. So Members (P, 
s), occurs before any time passes. If 7 delivers m, rcv; (m) is triggered during the Members; (P, s) 
action, so rcv;(m) also occurs before any time passes. | 


7.7 Extending Atom to Cope with Late Messages 


In this paper, we assumed a synchronous model with deterministic network latency guarantees. 
Since the network latency, A is expected to be of a smaller order of magnitude than 0, it would 
not significantly hurt time bounds if conservative assumptions are made in the choice of A. 

In ongoing research we are considering networks where latency bounds are more likely to be 
violated. For example, some networks may support differentiated services with probabilistic latency 
guarantees. Moreover, loss rates may exceed the bounds assumed in the implementation of the 
reliable network. Such networks can be represented using the timed-asynchronous [11] failure 
model. 

Although our algorithm cannot guarantee atomic broadcast semantics while network latency 
and reliability guarantees are violated, it is important for the algorithm to be able to recover 
from such situations, and to once more provide correct semantics after network guarantees are re- 
established. In addition, it would be desirable to inform the application when a violation of Atom 
semantics occurs, and when the correct semantics are resumed (following the failure awareness 
approach of [14]). 

There are some strategies that can be used to make Atom recover from periods in which network 
guarantees are violated. For example, a lost or late message can cause inaccurate failure suspicions. 
With Atom, if a process k is falsely suspected, it will receive a (CUP-INIT, k) message for itself. 
In order to recover from such a situation, we could have the process “commit suicide” in such a 
situation, that is inform the application of the failure and have the application re-join as a new 
process. The full modification of Atom for this setting is ongoing work. 


8 Conclusions 


We have defined two new problems, Dynamic Atomic Broadcast and Consensus with Unknown 
Participants. We have presented new algorithms for both problems. The latency of both of our 
algorithms depends linearly on the number of failures that occur during a particular execution, but 
does not depend on an upper bound on the potential number of failures, nor on the numbers of 
joins and leaves that happen during the execution. 
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A Correctness Proofs 


A.1 Correctness of the CUP Algorithm 


We consider the system consisting of a composition of automata CUP;, one for each iz € I. We 
consider a restricted set of executions of this composition—those in which the environment safety 
assumptions are all satisfied. The invariants we state throughout this section should be interpreted 
as saying that the stated property is true for all states that occur in such executions. 


A.1.1 General invariants 


We say that a message is in the Net if a net_mcast event for that message has occurred or is in a 
trigger-buffer. 
The first invariant lists an assortment of basic constraints. They can be proved using induction. 


Invariant A.1 1. value[r,i]; = 1 if and only if world[r,i]; = 1. 


2. If value[r,i]; = v #1 and world[r,i]; = W #1 then an (i,r,v,W) message is in the 
Net. 


. If Ci,r,*,*) is in the Net then round; > r. 
. If mode; = L then round; = 0. 
If mode; = running then some (i,1,*,*) message is in the Net. 


. Ifi € failed[r]; then fail; has occurred. 


2 DB A KR vw 


. [fi € failed[r]; ands > r, then value[s,i]; = L. 


The next invariant expresses consistency of values and worlds of the same process at different places 
in the system. 


Invariant A.2 1. If messages (j,r,v,W) and (j,r,v’,W’) are in the Net thenv = v’ andW 
= W. 


2. If value[r,j]; # Ll and value[r,jly # L then value[r,j]; = valuel[r, jj. 
8. If world[r,j]; A L and world[r,jly # L then world[r,jl]; = world[r, jly. 


4. If valuelr,j]; # L and world([r,j]; # L and a message (j,r,v,W) is in the Net then 
value[r,j]; = v and world[r,j];= W. 


The next two invariants describe some facts that follow from the existence of OUT messages and 
from the detection of leaves. 


Invariant A.3 1. ‘If an (i,OUT) message is in the Net then mode; = done. 


2. [fi € out[r]; then mode; = done. 
Proof: By induction. Part 2 uses the accurate leave detector assumption. | 


Invariant A.4 [fi € out[r];, ands > r, then no message of the form (i,8,*,*) is in the Net, 
and for all 7, value[s,i]; = 1. 
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Proof: By strong induction. First, we claim that a net_mcast; event cannot convert the invariant 
from true to false by falsifying the conclusion while leaving the hypothesis true. This is because, 
if the hypothesis is true, then 7 € out[r], in the pre-state of the net_mcast;, which implies, by 
Invariant A.3, that mode; = done. But the precondition of net_mcast; requires that mode; = 
running, a contradiction. 

The key steps are, therefore, those that make the hypothesis true. Index 7 can be added to 
out Lr], by receipt of an OUT message by & or by a leave_detect,(i). An OUT message may 
result from a previous abstain; that occurs when mode; = 1, or a previous decide; event. 

For abstain;, by Invariant A.1, we know that in the pre-state of the abstain,;, round; = 0. 
Then Invariant A.1 implies that in the pre-state, no message of the form (i,*,*,*) is in the Net, 
and for all j and all s, value[s,i]; = L. Once the abstain; happens, mode; becomes done, which 
means that no later messages are sent. 

For decide;, the FIFO assumption for message delivery implies that the decide; event must have 
occurred when round = r —1. Invariant A.1 then implies that in the pre-state of the decide;, the 
conclusion of the invariant holds. Since the decide; event sets mode; to done, 7 sends no further 
messages, so the conclusion continues to hold. 

For leave_detect, (i), we know by the lossless leave assumption that before the leave_detect, (i) 
occurs, k has already received every message that has ever been net_mcast by 7. Since k explicitly 
checks that it has no values from 27 for round r, there are no such messages in the Net. a 


The following says that any value that appears anywhere in the system is some participant’s initial 
value. 


Invariant A.5 1. If (i,r,v,W) is in the Net then there exists j and W’ such that (j,1,v,W’) 
is in the Net. 


2. If value[r,k]; = v #1 then there exists j and W’ such that (j,1,v,W’) is in the Net. 


Proof: We show Parts 1 and 2 together by induction on the length of a finite execution. 
Base: Trivial, because no messages are initially in the Net and no values are initially non-L. 
Inductive step: We first show part 1. The interesting steps are those in which a message (i,r,v,W) 
is put into the Net. If r =1 then (i,r,v,W) is put into the Net by an init;(v,W) event, which 
puts the net_mcast into trigger-buffer;. But this immediately satisfies the conclusion. On the 
other hand, ifr > 2, then (i,r,v,W) is put into the Net by an explicit net_mcast(i,r,v,W) step. 
In this case, v is obtained from a set of values already in i’s value array in the pre-state. The 
induction hypothesis, part 2, then implies that some (j,1,v,W’) is already in the Net, as needed. 
For part 2, the key step is a net_rcv;(k,r,v,W) for some W. In the pre-state of such a step, mes- 
sage (k,r,v,W) isin the Net. The inductive hypothesis, part 1, then implies that some (j,1,v,W’) 
is already in the Net, as needed. | 


The following invariant asserts that processes are always in their own worlds. 
Invariant A.6 1. If a message (i,r,v,W) is in the Net theni © W. 


2. If world[r,i]l; # 1 theni € world[r,i];. 
Proof: We prove part 1 by induction on the length of the execution, with a trivial base case. 


Inductive step: The interesting steps are those in which a message (i,r,v,W) is put into the Net. 
If r = 1, then this is done by an init;(v,W) step. In this case, the environment well-formedness 
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assumption implies that no leave; or fail; event precedes the init;, and so the world consistency 
assumption implies that 7 € W, as needed. On the other hand, ifr > 2, then (i,r,v,W) is put into 
the Net by an explicit net_mcast(i,r,v,W) step. In this case, the precondition says that mode; = 
running and world[1,i]; ¢ 1 in the pre-state. In this pre-state, 2 is not in any failed[r], set, 
because if it were, Invariant A.1 would imply that z has failed, and it would not be able to perform 
the net_mcast. Also, in this pre-state, 7 is not in any out [r], set, by Invariant A.3. Therefore, i 
is included in W, because of the way W is defined. 

Part 2 follows follows from part 1 and Invariant A.1. a 


The following invariant describes consequences of the definition of a round r+1 world and value: 


Invariant A.7 If (i,rt+1,v,W) is in the Net, forr > 1, then: 


1. For every j € W, world[r,j]; 4 L. 


2. W = world[r,i]; \ out[r]; \ failed[r];. 
3. v = min {value[r,j]; : 7 € Wh. 
4. W = world[1,i]; \ out-by[r],; \ failed-by[r]};. 


Proof: Part 1 is proved by an easy induction; the key step is net_mcast (i,r+1,v,W), and the 
conclusion follows immediately from the precondition. 

Given part 1, we prove part 2 by induction. Now the interesting steps are net_mcast;(i,r+1,v,W), 
net_rcv;(j,0UT), leave_detect;(j), and fail_detect;(j). The fact that net_mcast,;(i,r+1,v,W) 
yields the property follows immediately from the precondition. A net_rcv;(j,OUT) event could 
only falsify the property if 7 € W and the event puts 7 into out[r];. However, part 1 implies 
that world[r,j]; # L in the post-state, and hence value[r,j]; # in the post-state. But 
this would cause the post-state to violate Invariant A.4, a contradiction. A similar argument 
shows that leave_detect;(j) cannot falsify the property. Finally, fail_detect;(j) could only 
falsify the property if 7 € W and the event puts 7 into failed[r];. However, part 1 implies that 
world[r,j]; # 1 in the post-state, and hence value[r,j]; # L in the post-state. But this would 
cause the post-state to violate Invariant A.1, a contradiction. 

We prove part 3 by induction, using part 1. This time, the interesting steps are net_mcast;(i,rt+1,v,W) 
and net_rcv;(j,r,*,*). Again, the net_mcast;(i,r+1,v,W) step yields the property immedi- 
ately. A net_rcv;(j,r,v’,*) could only falsify the property if 7 € W. But in this case we know 
that value[r,j]; # 1 ithe pre-state, and then Invariant A.2 implies that v’ = value[r,j]; in 
the pre-state. It follows that this step does not change value[r,j];, and so does not falsify the 
property. 

Part 4 is proved by induction on r (not induction on the length of the execution), using part 
2. The base case, r = 1, follows immediately from part 2. For the inductive step, we sup- 
pose that the claim is true for some r > 1 and show it for r + 1. That is, we assume that 
(i,r+2,v,W) is in the Net. Then by part 2, W = world[rt1,i],; \ out[r+1]; \ failed[rt1]j. 
Now, since world[rt+1,i]; #4 1, Invariant A.1 implies that a message of the form (i,r+1,v’,W’) 
is in the Net, where W’? = world[r+1,i];. By inductive hypothesis, part 4, this implies that W’ 
= world[1,i]; \ out-by[r]; \ failed-by[r];. Therefore, W = world[1,i]; \ out-by[r]; \ 
failed-by[r]; \ out[r+1]; \ failed[rt1];. This is equal to world[1,i]; \ out-by[rt1]; \ 
failed-by [rt+1];, as needed. | 


Invariant A.8 Suppose that decide;(v) has happened at round r. Then: 
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1. For allj € world[r,i]; \ out[r];, value[r,j]; = value[r,i]; and world[r,j]; C world[r,i]j. 
2. For all j € world[r,il;, if (j,r,v’,W) is in the Net thenv’? = v andW C world[r,il,. 


Proof: Part 1 follows from an easy induction: out can only grow, and value and world do not 

change once they are non-L. Therefore, the only interesting step is decide;(v), and the result 
follows directly from the precondition. 

For part 2, consider any state s after a decide;(v) has happened at round r. Suppose that 

j € world[r,i]; and (j,r,v’,W) is in the Net, in state s. Then Invariant A.4 implies that 7 ¢ 

out [r];. Thus, 7 € world[r,i],;\ out[r];. Then part 1 and Invariant A.2 imply the conclusion. 

|_| 


The following invariants say that any process’ values and worlds decrease as rounds increase. 


Invariant A.9 For any r > 1, if a message (i,rt+1,v,W) is in the Net then value[r,i]l; #1, v 
< value[r,il]l;, andW C world[r,il],. 


Proof: By induction. For the inductive step, the interesting case is when the last action of the 
execution is net_mcast (i,rt+t1,v,W). Invariant A.6 implies that 7 € W. Therefore, the precondition 
for net_mcast (i,r+1,v,W) implies that, in the pre-state, value[r,i]; # L. Therefore, this is also 
true in the post-state, as needed. 

Next, we show that v < value[r,il];. The value v is determined in the net_mcast event to be 
the minimum of the set of values of the form value [r,j];, for 7 € W. Since 2 € W, this minimum 
includes value[r,i],;. Therefore, v < value[r,i]j;. 

Finally, we show that W C world[r,il;. The value W is determined in the net_mcast event to 
be world[r,i]; \ out-by[r] \ failed-by[r], according to the values of the out and failed sets 
in the pre-state. It follows immediately that Wis a subset of world[r,i];. | 


Invariant A.10 For any r > 1, 


1. Ifvalue[r+1,i]; # 1 then value[r,i]; # L, value[r+1,i]; < value[r,i];, andworld[rt1,i]; C 
world[r,i];,. 


2. If value[r,i]; # Ll andi < s < r then value[s,i]; # L, value[r,i]; < valuel[s,il],, 
and world[r,i]; C world[s,i]j;. 


Proof: For part 1, assume that, in some reachable state, value [r+1,i]; 4 L, and hence world[r+1,i]; 
# 1. Then Invariant A.1 implies that in the same state, a message (i,r+1,v,W) must be in the 
Net, where v = value[rt+1,i]; and W = world[rt+1,i];. Invariant A.9 then yields the conclu- 
sions. Part 2 follows from part 1, using induction on r-s. | 


The following invariant says that, if all the messages for a particular round r are “consistent”, then 
so are all the messages for all later rounds. 


Invariant A.11 Let W be a nonempty finite set, v € V, andr >1. Suppose that, for every i € W, 
if a message of the form (i,r,v’ ,W’) is in the Net, thenW’? C Wandv’ = v. 
Then for every 1,7 € W and for every s > r, 


1. If a message of the form (i,8s,v’ ,W’) is in the Net, then W’? C Wandv = v’. 


2. value[s,i]; is either v or L. 
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3. world[s,i]; is either a subset of W or L. 


Proof: We prove part 1 by induction on the length of an execution. 

Base: The conclusion of the invariant is vacuously true in the start state. 

Inductive step: The interesting steps are those that put some message (i,s,v’,W’) into the Net, 
wherei € Wands > r. We may restrict attention to the case where s > r, because ifs = r and 
the step falsifies part 1, it also falsifies the hypothesis of the invariant. Thus, the only interesting 
steps are of the form net_mcast(i,s,v’,W’) where? © Wands > r. So consider such a step, and 
fix i, s, v’, and W’. Assume that the hypothesis of the invariant is true after (and hence before) 
the step. 

We show that W’? C W. After the net_mcast step, the message is in the Net. Invariant A.9 
then implies that W’ C world[s-1,i];. Invariant A.10 then implies that world[r,i]; # 1 
and world[s-1,i]; C world[r,i];. Therefore, W’?’ C world[r,i];. Since world[r,i]; # L, 
an (i,r,*,W’’) message is in the Net, where W’’ = world[r,il];. Then since the hypothesis of 
the invariant is true, it follows that world[r,i]; C W. Putting all the pieces together yields that 
Ww? C W. 

Next, we argue that v’ = v. The value v’ is determined by the precondition of the net_mcast 
action, as the minimum of a set of values value [s-1,j1];, taken over all indices 7 in W’. Because W’ 
C W, every such index 7 is in W. Since value[s-1,j]; A 1, a (j,s-1,v’’,W’’) message is in the 
Net in the pre-state of the new net_mcast, where v’’ = value[s-1,j];. Our assumption that the 
conclusion of the invariant is true in the pre-state then implies that value[s-1,j]; = v. Thus, all 
the values considered in the min are equal to v, which implies that v = v’. 

This proves part 1. Parts 2 and 3 follow from part 1 and Invariant A.1. a 


Invariant A.12 Jf round; = r > 1 and mode; = running and j is not failed, then round; > r 
st. 


Proof: Since j is not failed, by Invariant A.1(6), 7 ¢failed[s],; for any s,so 7 ¢failed-by [r-1]j. 
By Invariant A.3(2), 7 ¢ out[s]; for any s, so 7 Gout-by[r-1];. Since mode; = running, j initi- 
ated, and by the world consistency assumption, 7 €world[1,i];. By Invariant A.7(4), 7 is in 7’s 
world for round r. Therefore, 7 must have received a round r-1 message from j before moving to 
round r. a 


A.1.2. CUP safety guarantees 


We now prove that the CUP implementation satisfies the CUP safety guarantees, assuming the 
environment satisfies the safety assumptions. 


Theorem A.1 The CUP algorithm satisfies well-formedness. 


Proof: This is straightforward from the code and the well-formedness assumptions on the en- 
vironment. For condition 1, assume that decide; occurs. Then in the preceding state, mode = 
running. mode is initially 1, and the only way it becomes running is via init;. So there must be 
a preceding init;. 

For condition 2, assume for the sake of contradiction that two decide; events occur. Part 1 
implies that an init; precedes the first decide;. The first decide; sets mode; to done. After this 
point, and before the second decide; event occurs, mode; must become running. This can happen 
only as a result of another init; event. This means that two init; events must occur, which 
contradicts the environment well-formedness assumption. Therefore, no more than one decide; 
event occurs. a 
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Theorem A.2 The CUP algorithm satisfies uniform agreement. 


Proof: If at most one decide event occurs, the result follows immediately. So assume that 
there are at least two decide events. Consider the first decide event, decide;(v). By the pre- 
condition, we know that in the pre-state, there exists r such that world[r,i]l; # 1 and Vj € 
world[r,i]; \ out[r],;, value[r,j]; = v and world[r,j]; C world[r,i];. Since 7 does not 
leave, abstain, or decide before the decide; event, we know that i ¢ out [r]; in the pre-state; there- 
fore, value[r,i]; = v. Also consider any particular later decide event, decide, (v’). As above, 
we know that in the pre-state of this event, there exists r’ such that world[r’,i’], # 1 and 
Vj € world[r’,i’], \ out[r’],, value[r’,j]y = v’ and world[r’,j]y C world[r’,i’];. 
Moreover, value[r’ ,i’], = v’. 

We now show that 7’ € world[r,i];. Since 7’ decides, it initiates and does not leave or fail before 
it decides. Since 7 initiates before it decides, and thus before 7’ decides, 7’ does not leave or fail 
before 7 initiates. Then the world consistency assumption implies that 7’ gets put into world[1,i]j. 
Ifr = 1 then we are done, so assume that r > 2. Then the value of world[r,iJ]; is determined in a 
net_mcast;(i,r,*,*) step. To see that 7’ is included in world[r,i];, note that that set is defined 
in the net_mcast;(i,r,*,*) step to include (at least) all processes in world[1,i]; that do not 
leave, abstain, decide, or fail before the net _mcast;(i,r,*,*) event. And 7’ does not leave, abstain, 
decide, or fail by then, because this net_mcast;(i,r,*,*) event happens prior to the decide;. 

We also know that 7’ ¢ out[r]; in the pre-state of decide;. This is because 7’ has not left, 
abstained, or decided before the decide,. 

Next, we show that r’ > r, that is, the round at which 7’ decides is at least as great as the 
round at which i decides. Since i’ € world[r,i]; and i’ ¢ out [r]; in the pre-state of decide;, the 
precondition for decide; implies that value[r,i’]; # L in the pre-state of decide;. This means 
that 7’ must send an (i’,r,v,*) message. This implies that the round r’ at which 7’ decides is at 
least as great as r, that is, r’ > r. 

Finally, we argue that v’ = v. Invariant A.8, part 2, implies that in the pre-state of decide;, 
if 7 € world[r,i],; and if (j,r,v’’,W’’) is in the Net, then v’’?’ = value[r,il]; and W’’ C 
world[r,i];. Since r’ > r and 7’ € world[r,il],;, Invariant A.11, part 2, implies that in the 
pre-state of decide,;, value[r’,i’], is either v or L. Since (as noted earlier) value[r’,i’?]; = 
v’, we have that v=v’. 

|_| 


Theorem A.3 The CUP algorithm satisfies validity. 


Proof: Part 1 follows from Invariant A.5. Part 2 follows from Invariant A.10. | 


A.1.3. CUP liveness guarantees 


We now show that CUP satisfies its liveness property—termination. Formally, the lemmas and 
theorem we state in this section should be interpreted with respect to an execution a of the 
composition of automata CUP; for 7 € J such that: 


1. All the environment safety and liveness assumptions are satisfied in a. 


2. ais “weakly fair” to all actions of all CUP; automata, in the sense that if an action is enabled 
from some point onward, it eventually is performed. 
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Lemma A.4 Let J be the set of processes that initiate and never decide, leave, or fail, and suppose 
thati € J. Ifinit;(v, W) occurs and 7 € W then either j € J or else j abstains, leaves, decides, or 
fails. 


Proof: Follows from the init occurrence assumption. | 


Lemma A.5 If process i initiates and never decides, leaves, or fails, then round; increases without 
bound. 


Proof: Let J be the set of all processes that initiate and never decide, leave, or fail. Assume for 
the sake of contradiction that, for some process 7 € J, round; is bounded. Let r be the smallest 
round number such that for some process 7 € J, round; is bounded by r, and fix such 7 € J. Process 
z cannot get stuck at round 0, because the init; action immediately increments the round to 1. So 
we may assume that r > 0. 

We argue that 7 cannot be stuck at round r, by showing that for some v, W, the net_mcast,;(i,rt+1,v,W) 
action is eventually enabled and stays enabled. Then weak fairness implies that net_mcast;(i,rt+t1,v,W) 
eventually occurs. 

We claim that the last precondition of net_mcast;(i,r+1,*,*) (the negation of the decide 
precondition) is always true. For if not, then decide;(v) would be enabled for some v, and would 
stay enabled forever. This implies, by weak fairness, that decide; occurs, a contradiction. 

Next, we claim that for every 7 € world[1,i], either 7 receives a round r message from j, or 
else i puts 7 into its failed[r’] set or out[r’] set for some r’ < r. Fix any such 7. Lemma A.4 
implies that either 7 € J or 7 eventually abstains, leaves, decides, or fails. If 7 € J then by choice 
of r, j does not get stuck at any round less than r, and so 7 eventually sends a round r message, 
which 7 eventually receives. 

If 7 fails, then eventually a fail_detect;(j) occurs, which makes 7 put j into one of its 
failed[r’] sets. If r’ < r then we are done; on the other hand, if r’ > r then 7 receives a 
round r message from j. 

If 7 abstains and does not fail, then eventually i puts 7 into its out[1] set (which suffices 
because 1 < r). If 7 leaves or decides at around r’ < r, then eventually 7 puts 7 into its out [r’] 
set. Finally, if 7 leaves or decides at a round r’ > r, then eventually 7 receives a round r message 
from j. 

This claim implies that eventually the precondition of net_mcast;(i,r+1,v,W) is satisfied for 
some v, W. Because the values and worlds can only decrease, eventually the precondition is satisfied, 
and remains satisfied, for the same v, W. Then weak fairness implies that the action eventually 
occurs, which moves j to round r + 1. This is a contradiction. | 


Lemma A.6 Let J be the set of processes that initiate and never decide, leave, or fail, and suppose 
that i € J. Then for r sufficiently large, world[r,i]; = J 


Proof: The result follows from two claims: that for allr, J C world[r,il;, and that for suffi- 
ciently large r, world[r,i]; is a subset of J. 

First, we show that for allr, J C world[r,il;. World consistency implies that J C world[1,i];. 
Since no element of J ever abstains, leaves, fails, or decides, no element of J is ever put into any 
failed[r],; or out[r];. Then the definition of world[r,i]; (in net_mcast (i,r,*,*)) implies that 
for allr, J C world[r,i];. 

Second, we show that for sufficiently large r, world[r,i]; is a subset of J. Let 7 be any element 
of world[r,i];. Lemma A.4 implies that if 7 ¢ J, then j eventually abstains, leaves, decides, or 
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fails. But in any of these cases, 7 eventually gets put into some failed[r]; or out [r];. This means 
that j is excluded from world[r,iJ]; for sufficiently large r. a 


Theorem A.7 The CUP algorithm satisfies termination. 


Proof: We prove that every process that initiates eventually decides, leaves, or fails. Assume for 
the sake of contradiction that there is at least one initiator that does not decide, leave, or fail. Let 
J be the set of processes that initiate and never decide, leave, or fail; then J is not empty. Then 
Lemma A.5 implies that the rounds of all processes in J increase without bound, and Lemma A.6 
implies that for sufficiently large r, world[r,i]; = J for alli € J. Thus from some round onward, 
every process in J bases its new value on values heard from exactly the members of J. 

Thereafter, each i € J eventually reaches some minimum value of value [r, i]; (by monotonicity 
and the fact that only finitely many values can be used). Consider a round beyond which all the 
minima have been attained. If these are all identical, then all processes can decide based on this 
value and world J, and we are done. On the other hand, if they are not all identical, then let 1 bea 
process whose minimum is larger than some other process’ minimum. Then 7 would see a smaller 
value and reduce its value further, a contradiction. | 


A.2 Atom Correctness Proof: Safety Arguments 
A.2.1 General Invariants 


The following invariants follow immediately from the code: 


Invariant A.13 Jf join-slot; oo then leave-slot; > join-sloty,. 


Invariant A.14 Suppose s € ended-slots;. Then: 
1. If j € joiners[s]; then join-slot; = s. 


2. If 7 € leavers[s]; then leave-slot,; = s. 


Proof: Process j can be inserted into joiners[s]; (leavers[s];) only if 2 receives a (j, JOIN, 
s-1) (resp. (j, LEAVE, s)) message, which can be sent only by j and only if join-slot; = s 
(resp. leave-slot; = s). = 


The following invariant asserts that from the join slot onward, slot messages (bulk, join, or 
leave) are multicast in order. 


Invariant A.15 Jf join-slot;—1< s’ < sands € mcast-slots; then s’ © mcast-slots;. 
Proof: join-slot; had to have been set before current-slot; becomes s’+1 because it is always 
chosen to be in the future. Therefore, net_mcast;(i, *, s’) is enabled once current-slot; 
becomes s’+1. This is earlier than the time at which net_mcast;(i, *, s) can occur, so time 


could not have passed beyond that point without net_mcast;(i, *, s-1) occurring. | 


The following invariant is central to the rest of the proof. It asserts that by the time of 
end_slot;(s), 2 has all the right processes in alive [s-1], alive[s], joiners[s], and joiners[s+1]. 
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Invariant A.16 If s € ended-slots; then 
1. If join-slot; < s ands € mcast-slots; then j € alive[s-1];U joiners[s]j. 


2. If join-slot; < s+1 and s+1 € mcast-slots; then j € alive[s];U joiners[s+1]. 


Proof: If 7 joined by slot s, it registered for the network before starting slot s-1. Moreover, 
ifs € mcast-slots;, then by Invariant A.15, s-1 € mcast-slots;, and therefore j multicasts 
either a (j, JOIN, s-1) or a bulk message in slot s-1, and by the Net’s reliable delivery property, 
this message is not lost due to j’s failure because 7 multicasts a message in the following slot, 
which occurs © time later, and we assume that © > A, and messages sent more than A time 
before the failure are not lost. Likewise, if join-slot; < s+1 and sti € mcast-slots,, then s 
€ mcast-slots,; (by Invariant A.15), and j multicasts a bulk or join message in slot s, which is 
not lost due to j’s failure. 

We will now show two things: first, that 7 joined early enough to get 7’s slot s-1 bulk or join 
message; and second, that end_slot;(s) occurred late enough for 7 to have received j’s slot s bulk 
or join message. 

Since 7 does end_slot for s, join-slot; < s. Process 7 chooses its join-slot following the 
net_join_OK; to be current-slot; + 2 + [I/O], so current-slot; becomes s-1 at least I time 
after the net_join_OK;. Since the maximum clock difference between 7 and 7 is I, j sends its 
message (join or bulk) for slot s-1 no earlier than the time of the net_join_OK;, so 7 joined early 
enough to get 7’s message for slot s. 

It is left to show that 7 gets j’s slot s bulk or join message for slot s before end_slot;(s). This 
follows from the precondition for end_slot; which asserts that clock; >(s+1)0+A+T. That 
is, that at least A +T time has elapsed since slot sti has begun at 7. Since the clock difference 
between 7 and 7 is at most I’, we get that at least A time has elapsed since slot st+1 has begun at 
j. Since j sends its slot s message once slot st+1 begins at 7, and the network latency is bounded 
by A, the message reaches 7 before end_slot;(s). | 


The following invariants are related to the suspects[s] sets. 

Invariant A.17 Jf suspects[s]; is not empty, then join-slot; < s. 

Proof: suspects[s]; gets set only upon end_slot;(s), for which this is a precondition. Once 

join-slot; is set to a non-oo value, it does not change, by the singularity of join and net_join_OK. 
a 

Invariant A.18 if 7 € suspects[s]; then j has failed. 

Proof: Since j gets inserted to suspects[s]; during end_slot;(s), j is in (alive[s-1]; U 

joiners[s]; \ leavers[s],;) \ alivel[s];). In particular, 7 is in alive[s-1],U joiners[s];, 

and therefore join-slot; < s. Moreover, j is not in alive[s];, so by the contrapositive of In- 

variant A.16(2), st1 ¢ mcast-slots,, which implies that j either fails or leaves before sending a 


slot s+1 message. Since 7 is also not in leavers[s];, 7 must have failed. | 


Invariant A.19 /f 7 € suspects[s]; andj € suspects[s’], then |s’ - s| <1. 
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Proof: Without loss of generality, assumes’ > s. Since j € suspects[s];, then 7 €Calive[s-1],U 
joiners [s];, and therefore join-slot; < s. Moreover, j isnot in alive[s]; whens € ended-slots;, 
so by the contrapositive of Invariant A.16(2), st1 ¢ mcast-slots;. By Invariant A.15, for any 
slot r > s,r ¢ mcast-slots,;, and therefore j ¢ alive[r], for any r > s. Since 7’ suspects j in 
slot s’, j is in alive[s’-1],, and therefore s’-1 < s. | 


The following invariant states that a process does not abstain from CUP instances pertaining 
to processes that it suspects. 


Invariant A.20 Jf k € suspects[s]; and CUP-status[k]; = done then CUP-dec-val[k]; # L. 


Proof: Assume by contradiction that the invariant is false. Since CUP-status[k]; = done while 
CUP-dec-val[k]; = L, then z must have performed CUP(k).abstain;. By the precondition for 
CUP (k) .abstain;, CUP-status [k]; was req when abstain; occurred, which implies that end_slot (s) ; 
could not have already occurred, that is, all the slots in ended-slots; were smaller than s 
at the time of CUP(k).abstain;. By the precondition for CUP(k) .abstain;, when it occurred, 
CUP-req-val[k]; had some non-l value, v, such that v < s-1. This, in turn, implies that a 
(CUP-INIT, &) message with slot v had previously arrived. That means that such a message was 
previously sent by some j, which implies that k is added to suspects[v];, during end_slot;(v), 
and remains there henceforward. But k € suspects[s]; and v < s-1, a contradiction to Invari- 
ant A.19. a 


Invariant A.21 Jfk € alive[s];,k ¢ alive[s];, ands € ended-slots; thenk € suspects[s];. 
Moreover, if sti € ended-slots,; then k € suspects[s+1];. 


Proof: Since & € alive[s];, we know that join-slot, < s < leave-slot, and that s € 
mcast-slots,. Therefore, by Invariant A.16(1), if s € ended-slots,; then k € alive[s-1],U 
joiners[s];. Additionally, k is neither in leavers[s]; nor in leavers[s];, because it does 
not leave at slot s. Therefore, since k ¢ alive[s];, in end_slot;(s), k gets inserted into 
suspects [s] j. 

Sincek ¢ alive[s],, by the contrapositive of Invariant A.16(2), we get that st1 ¢ mcast-slotsy,. 
That is, & does not send a bulk or leave message for slot st1. Therefore, k ¢ alive[s+1],U 
leavers[s+1]; when end_slot;(s+1) occurs, and k gets inserted into suspects[s+1]; when st+1 
is inserted to ended-slots;. | 


Invariant A.22 Ifk € alivel[s]; ands+1 € ended-slots; and CUP-dec-val[k]; < s thenk € 
suspects[st1]j;. 


Proof: Since k € alive[s];, join-slot, < s < leave-slot,. Since CUP-dec-val[k]; <, 
then by the validity property of CUP, some process | must have initiated CUP(k) with an ini- 
tial value s’ < s. This implies that k € suspects[s’]), and therefore k ¢ alivel[s’], and s’ 
€ ended-slots;. By contrapositive of Invariant A.16, s’+1 ¢ mcast-slots,;, and therefore also 
s’+1 ¢ mcast-slots;. So 7 does not hear a bulk or leave message from k for slot st1, and k € 
suspects[s+t1];. a 


Lemma A.8 Assume that for some processes 7,k,1 CUP(k).init,;(v, W) occurs with 7 EW, and 
that CUP(k) .init;(v’, W’) also occurs. Then net_join_OK; has occurred before CUP(k) .init;(v’, 
W’). 
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Proof: By the precondition for CUP(k).init, k € suspects[v’]; and k € suspects[v],, so by 
Invariant A.19, v'’ > v —1. When CUP(k).init;(v, W) occurs, W = alive[v],U joiners[v+t1]). 
Since j EW, this implies that join-slot; < vt1 < v’+2. Assume CUP(k).init;(v’, W’) occurs 
at time t. So at time ¢t, v’ € ended-slots;. By the precondition for end_slot;(v’), at time ¢ 
clock; > (v'+1)O+A+T. Since v’+1 > join-slot; —1, at this time clock; > (join-slot;— 
1)0+A+T. Since the clock skew is bounded by I’, at time t, clock; > (join-slot;—1)0+ A. 
So ¢ is at least A time after 7 begins slot join-slot; — 1. But join-slot; is chosen to be at 
least 2 slots after the slot at which net_join_OK; occurs at 7, so 7 begins join-slot; —1 after the 
net_join_0K;, i.e., before time t. | 


A.2.2 Safety environment conditions for CUP 


Well-formedness CUP(k).init; only occurs when k becomes suspected at 7. Once k is sus- 
pected, it is never again alive. Therefore, it is never suspected again and CUP(k).init,; occurs at 
most once. By Invariant A.20, since k is suspected at i, 7 does not abstain. Thus, at most one 
init; or abstain; event occurs. 

The fact that at most one leave; event occurs and at most one fail; event occurs is ensured 
by the application, since leave and fail actions are routed directly from the application to all 
instances of CUP. 

The fact that no fail; precedes an init; follows from the fact that failures affect all components 
and processes do not take any steps after they fail. 

One of the preconditions end_slot; is that leave-slot; = oo, that is, that leave; did not 
occur. Therefore, no leave; precedes an init;. 


World consistency Assume that CUP(k).init;(s, W) occurs at time t, 7 does not leave or fail 
before time ¢, and CUP(k).init;(s’, *) also occurs. We need to show that 7 EW. 

CUP(k).init;(s, W) is triggered during end_slot;(s). We need to show that at this time j € 
alive[s];U joiners[st1];. This is true if 7 receives 7’s slot s bulk or join message. 

By the precondition for end_slot;(s), clock; > (st1)O+A+T at time t. Since the difference 
between a process clock and real time is at most I'/2, the real time associated with point t is at 
least (st1)O0 + A+T/2. By assumption, j does not fail or leave until this time. 

By Invariant A.19, s’ < sti. When CUP(k).init;(s’,*) occurs, k €suspects[s’];. By 
Invariant A.17, join-slot; < s’. Together these two inequalities imply that join-slot; < st1. 
Therefore, if 7 does not fail or leave before clock; becomes s+10, j multicasts its slot s bulk or join 
message when clock; = (st1)© (a join message is multicast if join-slot; = s+1; otherwise j 
multicasts a slot s bulk message). When clock; = (s+1)0Q, the real time is at most (st1)O+T°/2. 
If j does not fail until the real time becomes (st1)0+T/2+ A, then j’s message is not lost, and i 
receives it by time (st+1)0+T/2+ A. But we assume that j does not fail or leave until this time. 


Accurate failure detector CUP(k).fail_detect,;(j) occurs only iffor some slot s 7 Gsuspects[s];. 
Therefore, by Invariant A.18, 7 has previously failed. Moreover, since a process is never again alive 
after it is suspected, it is never again suspected, and CUP(k) .fail_detect;(j) does not recur. 


Accurate leave detector CUP(k).leave_detect;(j) occurs only if a LEAVE message is re- 
ceived from 7; 7 sends at most one such a message and only if it actually leaves. 
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Lossless leave Assume a CUP process at j multicasts a message m, and subsequently, leave; 
occurs. When leave; occurs, a LEAVE message is inserted to out-buf; to be sent in the ensuing 
slot. This LEAVE message is multicast after m. leave_detect;(j) occurs when this LEAVE 
message is received. By the FIFO property of Net, net_rcv,;(m) occurs beforehand. 


A.2.3 Proving the total order property 


We now prove that all the process deliver messages in a consistent total order. We define the total 
order S as follows: Let P, be the union of all sets P such that an action members(P, s),; occurs. 
The set of messages S,, is defined to be those messages included in slot s bulk messages by processes 
in P,. The set of messages in S is defined to be the union of all sets Sj. 

The ordering is based on slots, so that for s < s’, all messages in S, precede all messages in 
S. For messages pertaining to the same set S;, the ordering is by process indices. For the same 
slot and process index, the ordering is the temporal order of sending (at the external boundary of 
Atom). 

We have to show that every process delivers a contiguous subsequence of S. We first prove 
Lemma A.9, asserting that every two processes that perform a members(P, s) action for a slot 
s do so with the same membership set P. As part of this action, processes deliver messages for 
slot s. Next, we prove Lemma A.10, asserting that if a process i performs members;(P, s) with 
j €P, then 7 has received a bulk message for slot s from j, and therefore triggers the delivery of 
all the messages included in it as an effect of the members;(P, s) action. Thus, every process 
that performs members(P, s), triggers the delivery of all the messages in S,. These messages are 
delivered in order of the sender’s process index, and for each process, in FIFO order. Therefore, 
these messages are delivered in the order defined on S,. 

Since every process performs members(P, s) for a contiguous subsequence of slots, every process 
delivers a contiguous subsequence of the messages in S. 

We now prove the lemmas: 


Lemma A.9 /[fmembers(P,s); and members(P’,s),; occur, then P= P’. 


Proof: Let k be a process in P. At the time members(P,s); occurs, k € alive[s];, sti € 
ended-slots;, and CUP-dec-val [k]; is either | or larger than s. Assume by way of contradiction 
that k ¢ P’, then either k ¢ alive[s]; or CUP-dec-val[k]; < s when members(P’ ,s); occurs. 

Assume first that k ¢ alive[s];. Note that s € ended-slots; when members(P’,s)j oc- 
curs, so by Invariant A.21, k € suspects[s]; at the time members(P’,s)j; occurs. By the pre- 
condition for members(P’,s);, CUP-status[k]; = done when it occurs, and by Invariant A.20, 
CUP-dec-val[k]; # 1, that is, CUP(k) .decide;(v) occurred for some v and set CUP-dec-val [k] ; 
= v. By the well-formedness property of CUP, j initiated CUP(k). Since k € suspects[s],, k 
cannot be included in suspects[s’]; for any s’ # s, and so j initiated CUP(k) with s. By the 
validity condition of CUP, v < s. 

Since sti € ended-slots; when members (P,s); occurs, by Invariant A.21,k € suspects[s+1]; 
at this time. Therefore, by the precondition for members (P,s);, CUP-status[k]; = done. By A.20, 
CUP-dec-val[k]; # L, that is, CUP(k) .decide; occurred, and by the uniform agreement property, 
CUP-dec-val[k]; = CUP-dec-val[k]; < s. A contradiction. 

Now, assume that CUP-dec-val[k]; < s whenmembers(P’,s),; occurs. Sinces+t1 € ended-slots; 
when members (P,s),; occurs, by Invariant A.22, k € suspects[st1]; at this time. Therefore, by 
the precondition for members (P,s);, CUP-status[k]; = done. By Invariant A.20, CUP-dec-val [k]; # 
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1, that is, CUP(k) .decide; occurred, and by the uniform agreement property, CUP-dec-val [k]; 
CUP-dec-val[k]; < s. A contradiction. 


Lemma A.10 Jf members;(P,s) occurs, then for every 7 EP, «1 received a bulk message for slot s 
from j prior to the members; (P,s) action. 


Proof: Assume members;(P,s) occurs. Since 7 €P, by the precondition for members;(P,s), 7 € 
alive[s],;. By definition of alive[s], in-buf[s,j] # 1 , that is, 7 received a bulk message from 
j for slot s. a 


A.3 Atom Correctness Proof: Liveness Arguments 


In the liveness proof, we can use the safety guarantees of CUP, since they depend only on the safety 
assumptions about the environment. 


A.3.1 General liveness lemmas 


Lemma A.11 Time passes. current-slot; increases through all slot values from zero onward, as 
long as i does not fail. 


Lemma A.12 [fi does not leave or fail, then end_slot;(s) occurs for every slots > join-slot;. 


Lemma A.13 [fi leaves and does not fail, then eventually 1 multicasts a (i, LEAVE, s) message. 


A.3.2. Liveness environment conditions for CUP 


Init occurrence Assume that init;(s, W) event occurs and 7 € W, and neither 7 nor j leaves or 
fails. 

Since j EW, 7 Calive[s] U joiners[st+1] at the time init;(s, W) is triggered, which means 
that net_join_0K; had already occurred prior to the init;(s, W) event. When init;(s, W) is 
triggered, i multicasts an (CUP-INIT, k) message. Since neither 7 nor 7 leaves or fails, j receives 
this message. 

Consider the pre-state value of CUP-status[k] when the (CUP-INIT, k) message from # arrives 
at j. If CUP-status[k] is running or done, then either CUP(k).init; or CUP(k).abstain; had 
to have already occurred and we are done. Otherwise, after this step CUP-status[k] = req, and 
CUP-req-val[k] =v. Since j does not leave or fail, by Lemma A.12, it eventually has slots larger 
than v in ended-slots;, so either CUP (k) .init;(*) or abstain CUP(k) .abstain; becomes enabled, 
depending on whether & is suspected in some slot or in none. 


Reliable delivery Assume that for some processes j,k,/ CUP(k).init,;(v, W) occurs with 7 EW, 
and that either CUP(k).init;(v’, W’) or CUP(k).abstain; occurs. We will show that by the 
time that either CUP(k).init;(v’, W’) or CUP(k).abstain; occurs, net_join_OK; had already 
occurred. This will imply that for any net_mcast;(m) that occurs after this event, a net rcv; (m) 
will occur unless either 7 will fail, or 7 will fail or leave. 

If CUP(k).init;(v’, W’) occurs, by Lemma A.8, net_join_0K; occurs first. Now, consider 
the case that CUP(k).abstain; occurs. Process i can only abstain after it receives an (CUP- 
INIT, k) message which could have only been sent if some other process 7’ has already triggered 
CUP (k).init,. By Lemma A.8, net_join_0K; must have occurred before the CUP (k) .init, event. 


42 


Complete leave and failure detector If CUP(k).init;(v,W) occurs with 7 € W, thenj €alivel[v]; 
U joiners[v+i]. Assume that 7 does not decide or leave or fail. Then CUP-status[k]; remains 
running from the time of the CUP (k) . init; (v,W) event onward. If leave; occurs, j sends a LEAVE 
message which 7 receives. When i receives j’s LEAVE message, 7 triggers leave_detect;(j) and 
we are done. Otherwise, assume j does not leave and fail; occurs, then eventually there is a 
slot for which i does not receive 7’s messages. Let s be the first such slot, so 7 ¢alivel[s], 
while 7 €alive[s-1]; U joiners[s];, so since j7 does not leave in s, 7 €suspects[s]. Since 
j €alive[v]; U joiners[vt1],s > v, andi triggers fail_detect;(j) while performing end_slot;(s). 


A.3.3  Liveness of Atom 


Eventual join Assume no fail; occurs. When join; occurs, net_join; is triggered, and by 
fairness, eventually occurs. By the eventual join property of Net, net_join_OK; eventually occurs. 
At that point, join-slot; is set to be bigger than current-slot;. join-slot, does not change from 
that point onward, since by the join integrity property of Net, no more net_join_OK; events occur. 
By Lemma A.11, current-slot; eventually becomes equal to join-slot;. When that happens, 
join_0K; becomes enabled, and remains enabled, as long as no time passes, until it occurs. By 
our assumption on time passage, no time passes until join_OK; occurs. Therefore, by fairness, it 
eventually occurs. 


Eventual leave Assume no fail; occurs. When leave; occurs, leave-slot; is set to be bigger 
than current-slot,;. leave-slot; does not change from that point onward, since by our assump- 
tion on the application, no more leave; events occur. By Lemma A.13, 7 eventually multicasts a (i, 
LEAVE, leave-slot,;) message, at which point leave-slot; is added to mcast-slots;. When that 
happens, net_leave; becomes enabled and remains enabled until it occurs. Then, by the eventual 
leave property of Net, net_leave_OK; eventually occurs and triggers leave_OK;. 


Message delivery The following lemma asserts that a process that participates in the algorithm 
and does not leave or fail continues to perform members(P, s) forever. 


Lemma A.14 /fmcast;(m) occurs for some m whens = current-slot,, and no fail; or leave; 
occurs, then for every s’ > s, members;(P, s’) occurs. 


Proof: Since mcast;(m) occurs, by our assumption about the application, it is preceded by a 
join_OK;. Therefore, m is appended to out-buf[s];. By Lemma A.11, current-slot; becomes 
st+1, and so 7 eventually sends its bulk message for slot s with m included in it. By liveness of Net, 
net_rcv;(i, m’, s) occurs, where m’ is 7’s slot s bulk message. 

By Lemma A.12, end_slot;(s) occurs for every slot s > join-slot,;. The sets suspects[s]; 
and suspects[s+1]; are set when end_slot;(s) (resp. end_slot;(s+1)) occurs, at which point 
a CUP instance for each process in these sets is initiated, and the set do not change afterwards. 
By the termination property of CUP, these instances of CUP eventually terminate, setting the 
corresponding CUP-status to done. Therefore, members(P, s),; eventually becomes enabled for 
some P, and by fairness, occurs. = 


We now prove that the message delivery liveness property holds. 


Assume mcast,;(m) occurs, and no fail; or leave; occurs. We first show that S contains m and 
rcev;(m) occurs. Let s = current-slot; when mcast;(m) occurs. By Lemma A.14, members(P, 
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s); occurs. We now show that i € P. This will imply that m € S (by definition of S), and that 
rcv; (m) occurs (since it is triggered by members(P, s);). 

To show that 7 €P, we have to show that j € alivel[s]; and CUP-dec-val[i]; = 1 at the 
time members(P, s),; occurs. Since at this time st1 € ended-slots;, by Invariant A.16, j € 
alive[s];. By Invariant A.18, since 7 does not fail it never becomes a suspect, and therefore, no 
instance of CUP is run for 2, and CUP-dec-val [i]; = L 

It remains to show that for every m’ that follows m in S, rcv;(m’) also occurs. By definition 
of S, m’ is included in a bulk message for some slot s’ > s from some process i’, such that i’ € P’ 
and members;(P’, s’) occurs for some j7. By Lemma A.14, members(P’’, s’); also occurs, and 
by Lemma A.9, P’? = P’?’. Therefore, rcv;(m’) is triggered by the members(P’’, s’); action. 
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